Decreasing image acquisition time for compressive imaging devices

ABSTRACT

Mechanisms for increasing the rate of acquisition of compressed/encoded image representations are disclosed. An imaging system may deliver subsets of a modulated light stream onto respective light sensing devices. The light sensing devices may be sampled in parallel. Samples from each light sensing device may be used to construct a respective sub-image of a final image. The parallelism allows compressed images to be acquired at a higher rate. The number of light sensing devices and/or the number of pixels per image may be selected to achieve a target image acquisition rate. In another embodiment, spatial portions of the incident light stream are separated and delivered to separate light modulators. In yet another embodiment, the incident light stream is split into a plurality of beams, each of which retains the image present in the incident light stream and is delivered to a separate light modulator.

RELATED APPLICATION DATA

This application claims the benefit of priority to U.S. ProvisionalApplication No. 61/372,826, filed on Aug. 11, 2010, entitled“Compressive Sensing Systems and Methods”, invented by Richard Baraniuk,Gary Woods, Kevin Kelly, Robert Bridge, Sujoy Chatterjee and LenoreMcMackin, which is hereby incorporated by reference in its entirety asthough fully and completely set forth herein.

This application is a continuation in part of U.S. application Ser. No.13/193,553, filed on Jul. 28, 2011 now U.S Pat. No. 8,570,405, inventedby Richard Baraniuk, Kevin Kelly, Robert Bridge, Sujoy Chatterjee andLenore McMackin, titled “Determining Light Level Variation inCompressive Imaging by Injecting Calibration Patterns into PatternSequence”, which is hereby incorporated by reference in its entirety asthough fully and completely set forth herein.

This application is also a continuation in part of U.S. application Ser.No. 13/193,556, filed on Jul. 28, 2011 now U.S Pat. No. 8,570,406,invented by Richard Baraniuk, Kevin Kelly, Robert Bridge, SujoyChatterjee and Lenore McMackin, titled “Low-Pass Filtering ofCompressive Imaging Measurements to Infer Light Level Variation”, whichis hereby incorporated by reference in its entirety as though fully andcompletely set forth herein.

This invention was made with Government support under Contract No.2009*0674524*000 awarded by the Central Intelligence Agency. TheGovernment has certain rights in the invention.

FIELD OF THE INVENTION

The present invention relates to the field of compressive sensing, andmore particularly to systems and methods for decreasing imageacquisition time in compressive imaging devices.

DESCRIPTION OF THE RELATED ART

According to Nyquist theory, a signal x(t) whose signal energy issupported on the frequency interval [−B,B] may be reconstructed fromsamples {x(nT)} of the signal x(t), provided the rate f_(S)=1/T_(S) atwhich the samples are captured is sufficiently high, i.e., provided thatf_(S) is greater than 2B. Similarly, for a signal whose signal energy issupported on the frequency interval [A,B], the signal may bereconstructed from samples captured with sample rate greater than B−A. Afundamental problem with any attempt to capture a signal x(t) accordingto Nyquist theory is the large number of samples that are generated,especially when B (or B−A) is large. The large number of samples istaxing on memory resources and on the capacity of transmission channels.

Nyquist theory is not limited to functions of time. Indeed, Nyquisttheory applies more generally to any function of one or more realvariables. For example, Nyquist theory applies to functions of twospatial variables such as images, to functions of time and two spatialvariables such as video, and to the functions used in multispectralimaging, hyperspectral imaging, medical imaging and a wide variety ofother applications. In the case of an image I(x,y) that depends onspatial variables x and y, the image may be reconstructed from samplesof the image, provided the samples are captured with sufficiently highspatial density. For example, given samples {I(nΔx,mΔy)} captured alonga rectangular grid, the horizontal and vertical densities 1/Δx and 1/Δyshould be respectively greater than 2B_(x) and 2B_(y), where B_(x) andB_(y) are the highest x and y spatial frequencies occurring in the imageI(x,y). The same problem of overwhelming data volume is experienced whenattempting to capture an image according to Nyquist theory. The moderntheory of compressive sensing is directed to such problems.

Compressive sensing relies on the observation that many signals (e.g.,images or video sequences) of practical interest are not onlyband-limited but also sparse or approximately sparse when representedusing an appropriate choice of transformation, for example, atransformation such as a Fourier transform, a wavelet transform or adiscrete cosine transform (DCT). A signal vector v is said to beK-sparse with respect to a given transformation T when thetransformation of the signal vector, Tv, has no more than K non-zerocoefficients. A signal vector v is said to be sparse with respect to agiven transformation T when it is K-sparse with respect to thattransformation for some integer K much smaller than the number L ofcomponents in the transformation vector Tv.

A signal vector v is said to be approximately K-sparse with respect to agiven transformation T when the coefficients of the transformationvector, Tv, are dominated by the K largest coefficients (i.e., largestin the sense of magnitude or absolute value). In other words, if the Klargest coefficients account for a high percentage of the energy in theentire set of coefficients, then the signal vector v is approximatelyK-sparse with respect to transformation T. A signal vector v is said tobe approximately sparse with respect to a given transformation T when itis approximately K-sparse with respect to the transformation T for someinteger K much less than the number L of components in thetransformation vector Tv.

Given a sensing device that captures images with N samples per image andin conformity to the Nyquist condition on spatial rates, it is often thecase that there exists some transformation and some integer K very muchsmaller than N such that the transform of each captured image will beapproximately K sparse. The set of K dominant coefficients may vary fromone image to the next. Furthermore, the value of K and the selection ofthe transformation may vary from one context (e.g., imaging application)to the next. Examples of typical transforms that might work in differentcontexts are the Fourier transform, the wavelet transform, the DCT, theGabor transform, etc.

Compressive sensing specifies a way of operating on the N samples of animage so as to generate a much smaller set of samples from which the Nsamples may be reconstructed, given knowledge of the transform underwhich the image is sparse (or approximately sparse). In particular,compressive sensing invites one to think of the N samples as a vector vin an N-dimensional space and to imagine projecting the vector v ontoeach vector in a series of M vectors {R(i)} in the N-dimensional space,where M is larger than K but still much smaller than N. Each projectiongives a corresponding real number s(i), e.g., according to theexpressions(i)=<v,R(i)>,where the notation <v,R(i)> represents the inner product (or dotproduct) of the vector v and the vector R(i). Thus, the series of Mprojections gives a vector U including M real numbers. Compressivesensing theory further prescribes methods for reconstructing (orestimating) the vector v of N samples from the vector U of M realnumbers. For example, according to one method, one should determine thevector x that has the smallest length (in the sense of the L₁ norm)subject to the condition that ΦTx=U, where Φ is a matrix whose rows arethe transposes of the vectors R(i), where T is the transformation underwhich the image is K sparse or approximately K sparse.

Compressive sensing is important because, among other reasons, it allowsreconstruction of an image based on M measurements instead of the muchlarger number of measurements N recommended by Nyquist theory. Thus, forexample, a compressive sensing camera would be able to capture asignificantly larger number of images for a given size of image store,and/or, transmit a significantly larger number of images per unit timethrough a communication channel of given capacity.

As mentioned above, compressive sensing operates by projecting the imagevector v onto a series of M vectors. As discussed in U.S. patentapplication Ser. No. 11/379,688 (published as 2006/0239336 and inventedby Baraniuk et al.) and illustrated in FIG. 1, an imaging device (e.g.,camera) may be configured to take advantage of the compressive sensingparadigm by using a digital micromirror device (DMD) 40. An incidentlightfield 10 passes through a lens 20 and then interacts with the DMD40. The DMD includes a two-dimensional array of micromirrors, each ofwhich is configured to independently and controllably switch between twoorientation states. Each micromirror reflects a corresponding portion ofthe incident light field based on its instantaneous orientation. Anymicromirrors in a first of the two orientation states will reflect theircorresponding light portions so that they pass through lens 50. Anymicromirrors in a second of the two orientation states will reflecttheir corresponding light portions away from lens 50. Lens 50 serves toconcentrate the light portions from the micromirrors in the firstorientation state onto a photodiode (or photodetector) situated atlocation 60. Thus, the photodiode generates a signal whose amplitude atany given time represents a sum of the intensities of the light portionsfrom the micromirrors in the first orientation state.

The compressive sensing is implemented by driving the orientations ofthe micromirrors through a series of spatial patterns. Each spatialpattern specifies an orientation state for each of the micromirrors. Theoutput signal of the photodiode is digitized by an A/D converter 70. Inthis fashion, the imaging device is able to capture a series ofmeasurements {s(i)} that represent inner products (dot products) betweenthe incident light field and the series of spatial patterns withoutfirst acquiring the incident light field as a pixelized digital image.The incident light field corresponds to the vector v of the discussionabove, and the spatial patterns correspond to the vectors R(i) of thediscussion above.

The incident light field may be modeled by a function I(x,y,t) of twospatial variables and time. Assuming for the discussion of thisparagraph that the DMD comprises a rectangular array, the DMD implementsa spatial modulation of the incident light field so that the light fieldleaving the DMD in the direction of the lens 50 might be modeled by{I(nΔx,mΔy,t)*M(n,m,t)}where m and n are integer indices, where I(nΔx,mΔy,t) represents theportion of the light field that is incident upon that (n,m)^(th) mirrorof the DMD at time t. The function M(n,m,t) represents the orientationof the (n,m)^(th) mirror of the DMD at time t. At sampling times, thefunction M(n,m,t) equals one or zero, depending on the state of thedigital control signal that controls the (n,m)^(th) mirror. Thecondition M(n,m,t)=1 corresponds to the orientation state that reflectsonto the path that leads to the lens 50. The condition M(n,m,t)=0corresponds to the orientation state that reflects away from the lens50.

The lens 50 concentrates the spatially-modulated light field{I(nΔx,mΔy,t)*M(n,m,t)}onto a light sensitive surface of the photodiode. Thus, the lens and thephotodiode together implement a spatial summation of the light portionsin the spatially-modulated light field:

${S(t)} = {\sum\limits_{n,m}{{I( {{n\;\Delta\; x},{m\;\Delta\; y},t} )}{{M( {n,m,t} )}.}}}$Signal S(t) may be interpreted as the intensity at time t of theconcentrated spot of light impinging upon the light sensing surface ofthe photodiode. The A/D converter captures measurements of S(t). In thisfashion, the compressive sensing camera optically computes an innerproduct of the incident light field with each spatial pattern imposed onthe mirrors. The multiplication portion of the inner product isimplemented by the mirrors of the DMD. The summation portion of theinner product is implemented by the concentrating action of the lens andalso the integrating action of the photodiode.

One of the challenges in compressive imaging is acquiring the samples ofthe signal S(t) fast enough to compete with FPA-based cameras that canoperate at video rates. (FPA is an acronym for “focal plane array”. Afocal plane array is an image-sensing device that includes an array oflight-sensing elements, typically rectangular, and typically positionedat the focal plane or image plane of an optical system.) For at leastsome values of N, an N-element FPA can acquire N-pixels images at videorates. If N is large, it may be difficult for the above-describedcompressive-imaging camera to acquire samples of the signal S(t) at arate sufficiently high to enable video capture, i.e., the capture ofencoded representations of N-pixel images at video rates. Thus, thereexists a need for mechanisms capable of increasing the rate at whichcompressive-imaging devices can acquire encoded images.

SUMMARY

In one set of embodiments, an imaging system may include a lightmodulation unit, two or more light sensing devices, an optical subsystemand a sampling subsystem.

The light modulation unit is configured to modulate an incident streamof light with a sequence of spatial patterns in order to produce amodulated light stream. The light modulation unit includes an array oflight modulating elements.

The optical subsystem is configured to deliver light from spatialsubsets (regions) of the modulated light stream onto respective ones ofthe light sensing devices. In other words, each light sensing devicereceives light primarily from the respective spatial subset of themodulated light stream. The spatial subsets of the modulated lightstream are produced by respective subsets of the array of lightmodulating elements. Each subset of the array of light modulatingelements produces the respective spatial subset of the modulated lightstream by modulating a respective spatial subset of the incident lightstream. The light sensing devices are configured to generate respectiveelectrical signals, where each of the electrical signals representsintensity of the respective spatial subset of the modulated light streamas a function of time.

The sampling subsystem is configured to acquire samples of theelectrical signals, preferably in parallel. The samples include samplesubsets that correspond respectively to the electrical signals. In otherwords, each sample subset comprises samples of the respective electricalsignal. The sample subsets are usable to reconstruct respectivesub-images of a final image that represents the incident light stream.Because the sub-images are smaller than the final image, the number ofsamples of each electrical signal needed to reconstruct the respectivesub-image is significantly smaller than the number of samples asingle-pixel camera would have to acquire to reconstruct the finalimage. Thus, the time required to acquire the compressed data-equivalentof the final image is significantly reduced. In video applications, thismeans that the imaging system may acquire compressed images at a higherframe rate than a single-pixel camera.

In another set of embodiments, an imaging system may include apre-modulation optical subsystem, a plurality of light modulation units,a plurality of light sensing devices, and a plurality ofanalog-to-digital converters (ADCs).

The pre-modulation optical subsystem is configured to receive anincident light stream and separate the incident light stream into aplurality of light substreams. The light substreams correspondrespectively to non-overlapping regions of the incident light stream.Thus, the incident light stream is divided spatially.

The light modulation units are configured to respectively modulate thelight substreams in order to generate respective modulated lightsubstreams. Each of the light modulation units is configured to modulatethe respective light substream with a respective sequence of spatialpatterns in order to produce the respective modulated light substream.Each of the light modulation units includes a respective array of lightmodulating elements.

The light sensing devices are configured to respectively receive themodulated light substreams and generate respective electrical signals.The electrical signal generated by each light sensing device representsintensity of the respective modulated light substream as a function oftime.

The ADCs are configured to sample the electrical signals, preferably inparallel. Each ADC is configured to configured to capture a sequence ofsamples of the electrical signal generated by a respective one of thelight sensing devices. The sample sequence captured by each ADC isusable to construct a respective sub-image representing a respective oneof the regions of the incident light stream. The sub-image may be joinedtogether to form the final image.

The imaging system may also include a plurality of post-modulationoptical subsystems configured to direct light from respective ones ofthe modulated light substreams onto respective ones of the light sensingdevices.

Various additional embodiments are described in U.S. ProvisionalApplication No. 61/372,826, which is hereby incorporated by reference inits entirety as though fully and completely set forth herein.

BRIEF DESCRIPTION OF THE DRAWINGS

A better understanding of the present invention can be obtained when thefollowing detailed description of the preferred embodiments isconsidered in conjunction with the following drawings.

FIG. 1 illustrates a compressive sensing camera according to the priorart.

FIG. 2A illustrates one embodiment of a system 100 that is operable tocapture compressive imaging samples and also samples of background lightlevel. (LMU is an acronym for “light modulation unit”. MLS is an acronymfor “modulated light stream”. LSD is an acronym for “light sensingdevice”.)

FIG. 2B illustrates an embodiment of system 100 that includes aprocessing unit 150.

FIG. 2C illustrates an embodiment of system 100 that includes an opticalsubsystem 105 to focus received light L onto the light modulation unit110.

FIG. 2D illustrates an embodiment of system 100 that includes an opticalsubsystem 117 to direct or focus or concentrate the modulated lightstream MLS onto the light sensing device 130.

FIG. 2E illustrates an embodiment where the optical subsystem 117 isrealized by a lens 117L.

FIG. 2F illustrates an embodiment of system 100 that includes a controlunit that is configured to supply a series of spatial patterns to thelight modulation unit 110.

FIG. 3A illustrates system 200, where the light modulation unit 110 isrealized by a plurality of mirrors (collectively referenced by label110M).

FIG. 3B shows an embodiment of system 200 that includes the processingunit 150.

FIG. 4 shows an embodiment of system 200 that includes the opticalsubsystem 117 to direct or focus or concentrate the modulated lightstream MLS onto the light sensing device 130.

FIG. 5A shows an embodiment of system 200 where the optical subsystem117 is realized by the lens 117L.

FIG. 5B shows an embodiment of system 200 where the optical subsystem117 is realized by a mirror 117M and lens 117L in series.

FIG. 5C shows another embodiment of system 200 that includes a TIR prismpair 107.

FIG. 6 shows one embodiment of system 600 for parallelizing theacquisition of a compressed representation of an image.

FIG. 7 show how the sample subsets acquired by system 600 may be used toconstruct respective sub-images of a complete image, where the completeimage represents the incident light stream.

FIG. 8 shows an embodiment of system 600 that includes the processingunit 150.

FIG. 9 illustrates an embodiment of system 600 that includes oneanalog-to-digital converter (ADC) per light sensing device (LSD).

FIG. 10 illustrates an embodiment of system 600 that includes aplurality of lenses to direct modulated light to respective ones of thelight sensing devices.

FIG. 11 illustrates an embodiment of system 600 including a 2×2 array oflenses (or lenslets) that directs modulated light onto a 2×2 array oflight sensing elements.

FIG. 12 illustrates an embodiment of system 600 including relay optics620R0, lenslet array 620LA and sensor array 630SA.

FIG. 13 illustrates an example of the electrical signals generated bythe light sensing devices in one embodiment.

FIG. 14 illustrates how four quadrants of the light modulating array (atleft) may be mapped to four corresponding light sensing devices, and thecorresponding sample subsets may be used to construct four correspondingsub-images, which being joined together represent the image carried bythe incident light stream (not shown; see FIG. 6).

FIG. 15 shows an embodiment of system 600 that includes a fiber couplingdevice 620FC and a plurality 620FB of fiber bundles to deliver spatialsubsets of the modulated light stream to respective light sensingdevices.

FIG. 16 shows an embodiment of the system 600 that include a TIR prismpair 1220 and digital micromirror device 110M.

FIG. 17A illustrates an embodiment of system 600 that include aplurality of light pipes to deliver spatial subsets of the modulatedlight stream to respective light sensing devices.

FIG. 17B illustrates one embodiment of the light pipes, in two differentviews.

FIG. 18 illustrates an embodiment of system 600 where lens 620FLconcentrates the modulated light stream onto the light sensing devices630.

FIG. 19 illustrates an embodiment of system 600 where lens 620FLconcentrates the modulated light stream onto the light sensing array630A.

FIG. 20 illustrates an embodiment of a circuit for reading rows ofsamples from an array of light sensing devices.

FIG. 21 illustrates an embodiment of a circuit for reading samples froman array of light sensing devices.

FIGS. 22A through 22F illustrates different ways of separating themodulated light stream into spatial portions.

FIG. 23 illustrates an embodiment of a method for parallelizing theacquisition of a compressed image representation.

FIG. 24 illustrates one embodiment of a system 2400 including one ADCper light sensing device.

FIG. 25 illustrates an embodiment of a system 2500 including circuitry2540, which is configured to read groups of samples from the lightsensing array 2530.

FIG. 26A illustrates one embodiment of a system 2600 for performingcompressive imaging.

FIG. 26B is a table illustrating the number of image frames per secondattainable from the system 2500 for different values of group rate, andnumber of light modulating elements (e.g., micromirrors) per FPAelement. FPA is an acronym for focal plane array.

FIGS. 27A and 27B illustrate embodiments of methods for designing acompressive-imaging device in order meet a target effective rate ofimage acquisition, especially for compressive-imaging devices that areto include a dedicated A/D converter per light sensing device.

FIGS. 28A and 28B illustrate embodiments of methods for designing acompressive-imaging device in order meet a target effective rate ofimage acquisition, especially for compressive-imaging device that are toinclude a focal plane array.

FIG. 29A illustrates one embodiment of a system 2900 that involvespatially separating the incident light beam to a number of sub-streamsprior to modulation.

FIG. 29B illustrates an embodiment of system 2900 where the incidentbeam is separated into two sub-streams.

FIG. 30A shows a mirror 2910M being used to separate the incident lightstream into two sub-streams.

FIG. 30B shows mirrors 2910Ma and 2910Mb being used to separate theincident light stream into three sub-streams.

FIG. 31A shows an angled mirror being used to separate the incidentlight stream into two sub-streams.

FIGS. 31B and 31C show two different ways prisms might be used toseparate the incident light stream into two sub-streams.

FIG. 32A shows an embodiment of an object 2910C having reflective faces(or surfaces) that meet at a point P, being used to separate theincident light stream into three sub-streams.

FIG. 32B shows the pattern of separation as seen along a central axis ofthe incident light beam, according to one embodiment.

FIG. 33 shows one embodiment of a method 3300 for parallelizing theacquisition of compressive-sensing samples for an image.

FIG. 34A illustrates one embodiment of a system 3400 that splits theincident light stream into two streams that contain the same imageinformation, and parallelizes the acquisition of samples for the commonimage.

FIG. 34B illustrates one embodiment of the optical separation unit 3410,as a beam splitter.

FIG. 35 illustrates one embodiment of a system 3500 that splits theincident light stream into G streams that contain the same imageinformation, and parallelizes the acquisition of samples for the commonimage.

FIG. 36 illustrates one embodiment of a computer system 3600 configuredto execute computer programs, e.g., programs implementing any of thevarious methods described herein.

While the invention is susceptible to various modifications andalternative forms, specific embodiments thereof are shown by way ofexample in the drawings and are herein described in detail. It should beunderstood, however, that the drawings and detailed description theretoare not intended to limit the invention to the particular formdisclosed, but on the contrary, the intention is to cover allmodifications, equivalents and alternatives falling within the spiritand scope of the present invention as defined by the appended claims.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The following patent applications provide teachings regardingcompressive sensing and compressive imaging:

U.S. Provisional Application No. 60/673,364 entitled “Method andApparatus for Optical Image Compression,” filed on Apr. 21, 2005;

U.S. Provisional Application No. 60/679,237 entitled “Method andApparatus for Reconstructing Data from Multiple Sources,” filed on May10, 2005;

U.S. Provisional Application No. 60/729,983 entitled “Random Filters forCompressive Sampling and Reconstruction,” filed on Oct. 25, 2005;

U.S. Provisional Application No. 60/732,374 entitled “Method andApparatus for Compressive Sensing for Analog-to-Information Conversion,”filed on Nov. 1, 2005;

U.S. Provisional Application No. 60/735,616 entitled “Method andApparatus for Distributed Compressed Sensing,” filed on Nov. 10, 2005;

U.S. Provisional Application No. 60/759,394 entitled “Sudocodes:Efficient Compressive Sampling Algorithms for Sparse Signals,” filed onJan. 16, 2006;

U.S. patent application Ser. No. 11/379,688 entitled “Method andApparatus for Compressive Imaging Device”, filed on Apr. 21, 2006; and

U.S. patent application Ser. No. 12/791,171 entitled “Method andApparatus for Compressive Imaging Device”, filed on Jun. 1, 2010.

Terminology

A memory medium is a non-transitory medium configured for the storageand retrieval of information. Examples of memory media include: variouskinds of semiconductor-based memory such as RAM and ROM; various kindsof magnetic media such as magnetic disk, tape, strip and film; variouskinds of optical media such as CD-ROM and DVD-ROM; various media basedon the storage of electrical charge and/or any of a wide variety ofother physical quantities; media fabricated using various lithographictechniques; etc. The term “memory medium” includes within its scope ofmeaning the possibility that a given memory medium might be a union oftwo or more memory media that reside at different locations, e.g., ondifferent chips in a system or on different computers in a network. Amemory medium is typically computer-readable, e.g., is capable of beingread by a computer.

A computer-readable memory medium may be configured so that it storesprogram instructions and/or data, where the program instructions, ifexecuted by a computer system, cause the computer system to perform amethod, e.g., any of a method embodiments described herein, or, anycombination of the method embodiments described herein, or, any subsetof any of the method embodiments described herein, or, any combinationof such subsets.

A computer system is any device (or combination of devices) having atleast one processor that is configured to execute program instructionsstored on a memory medium. Examples of computer systems include personalcomputers (PCs), workstations, laptop computers, tablet computers,mainframe computers, server computers, client computers, network orInternet appliances, hand-held devices, mobile devices, personal digitalassistants (PDAs), computer-based television systems, grid computingsystems, wearable computers, computers implanted in living organisms,computers embedded in head-mounted displays, computers embedded insensors forming a distributed network, etc.

A programmable hardware element (PHE) is a hardware device that includesmultiple programmable function blocks connected via a system ofprogrammable interconnects. Examples of PHEs include FPGAs (FieldProgrammable Gate Arrays), PLDs (Programmable Logic Devices), FPOAs(Field Programmable Object Arrays), and CPLDs (Complex PLDs). Theprogrammable function blocks may range from fine grained (combinatoriallogic or look up tables) to coarse grained (arithmetic logic units orprocessor cores).

As used herein, the term “light” is meant to encompass within its scopeof meaning any electromagnetic radiation whose spectrum lies within thewavelength range [λ_(L), λ_(U)], where the wavelength range includes thevisible spectrum, the ultra-violet (UV) spectrum, infrared (IR) spectrumand the terahertz (THz) spectrum. Thus, for example, visible radiation,or UV radiation, or IR radiation, or THz radiation, or any combinationthereof is “light” as used herein.

In some embodiments, a computer system may be configured to include aprocessor (or a set of processors) and a memory medium, where the memorymedium stores program instructions, where the processor is configured toread and execute the program instructions stored in the memory medium,where the program instructions are executable by the processor toimplement a method, e.g., any of the various method embodimentsdescribed herein, or, any combination of the method embodimentsdescribed herein, or, any subset of any of the method embodimentsdescribed herein, or, any combination of such subsets.

System 100 for Operating on Light

A system 100 for operating on light may be configured as shown in FIG.2A. The system 100 may include a light modulation unit 110, a lightsensing device 130 and an analog-to-digital converter (ADC) 140.

The light modulation unit 110 is configured to modulate a receivedstream of light L with a series of spatial patterns in order to producea modulated light stream (MLS). The spatial patterns of the series maybe applied sequentially to the light stream so that successive timeslices of the light stream are modulated, respectively, with successiveones of the spatial patterns. (The action of sequentially modulating thelight stream L with the spatial patterns imposes the structure of timeslices on the light stream.) The light modulation unit 110 includes aplurality of light modulating elements configured to modulatecorresponding portions of the light stream. Each of the spatial patternsspecifies an amount (or extent or value) of modulation for each of thelight modulating elements. Mathematically, one might think of the lightmodulation unit's action of applying a given spatial pattern asperforming an element-wise multiplication of a light field vector(x_(ij)) representing a time slice of the light stream L by a vector ofscalar modulation values (m_(ij)) to obtain a time slice of themodulated light stream: (m_(ij))*(x_(ij))=(m_(ij)*x_(ij)). The vector(m_(ij)) is specified by the spatial pattern. Each light modulatingelement effectively scales (multiplies) the intensity of itscorresponding stream portion by the corresponding scalar factor.

The light modulation unit 110 may be realized in various ways. In someembodiments, the LMU 110 may be realized by a plurality of mirrors(e.g., micromirrors) whose orientations are independently controllable.In another set of embodiments, the LMU 110 may be realized by an arrayof elements whose transmittances are independently controllable, e.g.,as with an array of LCD shutters. An electrical control signal suppliedto each element controls the extent to which light is able to transmitthrough the element. In yet another set of embodiments, the LMU 110 maybe realized by an array of independently-controllable mechanicalshutters (e.g., micromechanical shutters) that cover an array ofapertures, with the shutters opening and closing in response toelectrical control signals, thereby controlling the flow of lightthrough the corresponding apertures. In yet another set of embodiments,the LMU 110 may be realized by a perforated mechanical plate, with theentire plate moving in response to electrical control signals, therebycontrolling the flow of light through the corresponding perforations. Inyet another set of embodiments, the LMU 110 may be realized by an arrayof transceiver elements, where each element receives and thenretransmits light in a controllable fashion. In yet another set ofembodiments, the LMU 110 may be realized by a grating light valve (GLV)device. In yet another embodiment, the LMU 110 may be realized by aliquid-crystal-on-silicon (LCOS) device.

In some embodiments, the light modulating elements are arranged in anarray, e.g., a two-dimensional array or a one-dimensional array. Any ofvarious array geometries are contemplated. For example, in someembodiments, the array is a square array or rectangular array. Inanother embodiment, the array is hexagonal. In some embodiments, thelight modulating elements are arranged in a spatially random fashion.

The light sensing device 130 is configured to receive the modulatedlight stream MLS and to generate an analog electrical signal I_(MLS)(t)representing intensity of the modulated light stream as a function oftime.

The light sensing device 130 may include one or more light sensingelements. The term “light sensing element” may be interpreted as meaning“a transducer between a light signal and an electrical signal”. Forexample, a photodiode is a light sensing element. In various otherembodiments, light sensing elements might include devices such asmetal-semiconductor-metal (MSM) photodetectors, phototransistors,phototubes and photomultiplier tubes.

In some embodiments, the light sensing device 130 includes one or moreamplifiers, e.g., transimpedance amplifiers (TIAs), to amplify theanalog electrical signals generated by the one or more light sensingelements.

The ADC 140 acquires a sequence of samples {I_(MLS)(k)} of the analogelectrical signal I_(MLS)(t). Each of the samples may be interpreted asan inner product between a corresponding time slice of the light streamL and a corresponding one of the spatial patterns. The set of samples{I_(MLS)(k)} comprises an encoded representation, e.g., a compressedrepresentation, of an image (or a video sequence) and may be used toconstruct the image (or video sequence) based on any constructionalgorithm known in the field of compressive sensing. (For video sequenceconstruction, the samples may be partitioned into contiguous subsets,and then the subsets may be processed to construct correspondingimages.)

In some embodiments, the samples {I_(MLS)(k)} may be used for somepurpose other than, or in addition to, image (or video) construction.For example, system 100 (or some other system) may operate on thesamples {I_(MLS)(k)} to perform an inference task, such as detecting thepresence of a signal or object, identifying a signal or an object,classifying a signal or an object, estimating one or more parametersrelating to a signal or an object, tracking a signal or an object, etc.In some embodiments, an object under observation by system 100 may beidentified or classified by virtue of its sample set {I_(MLS)(k)}, orparameters derived from that sample set, being similar to one of acollection of stored sample sets (or parameter sets).

In some embodiments, the light sensing device 130 may include anamplifier, e.g., a transimpedance amplifier (TIA), to amplify the analogelectrical signal prior to A/D conversion.

In some embodiments, the light sensing device 130 includes exactly onelight sensing element. For example, the single light sensing element maybe a photodiode.

In some embodiments, the light sensing device 130 may include aplurality of light sensing elements (e.g., photodiodes). Each lightsensing element may convert light impinging on its light sensing surfaceinto a corresponding analog electrical signal representing intensity ofthe impinging light as a function of time. In some embodiments, eachlight sensing element may couple to a corresponding amplifier so thatthe analog electrical signal produced by the light sensing element canbe amplified prior to digitization. System 100 may be configured so thateach light sensing element receives, e.g., a corresponding spatialportion (or spectral portion) of the modulated light stream.

In one embodiment, the analog electrical signals produced, respectively,by the light sensing elements may be summed to obtain a sum signal. Thesum signal may then be digitized by the ADC 140 to obtain the sequenceof samples {I_(MLS)(k)}. In another embodiment, the analog electricalsignals may be individually digitized, each with its own ADC, to obtaincorresponding sample sequences. The sample sequences may then be addedto obtain the sequence {I_(MLS)(k)}. In another embodiment, the analogelectrical signals produced by the light sensing elements may be sampledby a smaller number of ADCs than light sensing elements through the useof time multiplexing. For example, in one embodiment, system 100 may beconfigured to sample two or more of the analog electrical signals byswitching the input of an ADC among the outputs of the two or morecorresponding light sensing elements at a sufficiently high rate.

In some embodiments, the light sensing device 130 may include an arrayof light sensing elements. Arrays of any of a wide variety of sizes,configurations and material technologies are contemplated. In oneembodiment, the light sensing device 130 includes a focal plane arraycoupled to a readout integrated circuit. In one embodiment, the lightsensing device 130 may include an array of cells, where each cellincludes a corresponding light sensing element and is configured tointegrate and hold photo-induced charge created by the light sensingelement, and to convert the integrated charge into a corresponding cellvoltage. The light sensing device may also include (or couple to)circuitry configured to sample the cell voltages using one or more ADCs.

In some embodiments, the light sensing device 130 may include aplurality (or array) of light sensing elements, where each light sensingelement is configured to receive a corresponding spatial portion of themodulated light stream, and each spatial portion of the modulated lightstream comes from a corresponding sub-region of the array of lightmodulating elements. (For example, the light sensing device 130 mayinclude a quadrant photodiode, where each quadrant of the photodiode isconfigured to receive modulated light from a corresponding quadrant ofthe array of light modulating elements. As another example, the lightsensing element 130 may include a bi-cell photodiode.) Each lightsensing element generates a corresponding signal representing intensityof the corresponding spatial portion as a function of time. Each signalmay be digitized (e.g., by a corresponding ADC) to obtain acorresponding sequence of samples. Each sequence of samples may beprocessed to recover a corresponding sub-image. The sub-images may bejoined together to form a whole image.

In some embodiments, the light sensing device 130 includes a smallnumber of light sensing elements (e.g., in respective embodiments, one,two, less than 8, less than 16, less the 32, less than 64, less than128, less than 256). Because the light sensing device of theseembodiments includes a small number of light sensing elements (e.g., farless than the typical modern CCD-based or CMOS-based camera), an entityconfiguring any of these embodiments may afford to spend more per lightsensing element to realize features that are beyond the capabilities ofmodern array-based image sensors of large pixel count, e.g., featuressuch as higher sensitivity, extended range of sensitivity, new range(s)of sensitivity, extended dynamic range, higher bandwidth/lower responsetime. Furthermore, because the light sensing device includes a smallnumber of light sensing elements, an entity configuring any of theseembodiments may use newer light sensing technologies (e.g., based on newmaterials or combinations of materials) that are not yet mature enoughto be manufactured into focal plane arrays (FPA) with large pixel count.For example, new detector materials such as super-lattices, quantumdots, carbon nanotubes and graphene can significantly enhance theperformance of IR detectors by reducing detector noise, increasingsensitivity, and/or decreasing detector cooling requirements.

In one embodiment, the light sensing device 130 is a thermo-electricallycooled InGaAs detector. (InGaAs stands for “Indium Gallium Arsenide”.)In other embodiments, the InGaAs detector may be cooled by othermechanisms (e.g., liquid nitrogen or a Sterling engine). In yet otherembodiments, the InGaAs detector may operate without cooling. In yetother embodiments, different detector materials may be used, e.g.,materials such as MCT (mercury-cadmium-telluride), InSb (IndiumAntimonide) and VOx (Vanadium Oxide).

In different embodiments, the light sensing device 130 may be sensitiveto light at different wavelengths or wavelength ranges. In someembodiments, the light sensing device 130 may be sensitive to light overa broad range of wavelengths, e.g., over the entire visible spectrum orover the entire range [λ_(L),λ_(U)] as defined above.

In some embodiments, the light sensing device 130 may include one ormore dual-sandwich photodetectors. A dual sandwich photodetectorincludes two photodiodes stacked (or layered) one on top of the other.

In one embodiment, the light sensing device 130 may include one or moreavalanche photodiodes.

In some embodiments, a filter may be placed in front of the lightsensing device 130 to restrict the modulated light stream to a specificrange of wavelengths or polarization. Thus, the signal I_(MLS)(t)generated by the light sensing device 130 may be representative of theintensity of the restricted light stream. For example, by using a filterthat passes only IR light, the light sensing device may be effectivelyconverted into an IR detector. The sample principle may be applied toeffectively convert the light sensing device into a detector for red orblue or green or UV or any desired wavelength band, or, a detector forlight of a certain polarization.

In some embodiments, system 100 includes a color wheel whose rotation issynchronized with the application of the spatial patterns to the lightmodulation unit. As it rotates, the color wheel cyclically applies anumber of optical bandpass filters to the modulated light stream MLS.Each bandpass filter restricts the modulated light stream to acorresponding sub-band of wavelengths. Thus, the samples captured by theADC 140 will include samples of intensity in each of the sub-bands. Thesamples may be de-multiplexed to form separate sub-band sequences. Eachsub-band sequence may be processed to generate a corresponding sub-bandimage. (As an example, the color wheel may include a red-pass filter, agreen-pass filter and a blue-pass filter to support color imaging.)

In some embodiments, the system 100 may include a memory (or a set ofmemories of one or more kinds).

In some embodiments, system 100 may include a processing unit 150, e.g.,as shown in FIG. 2B. The processing unit 150 may be a digital circuit ora combination of digital circuits. For example, the processing unit maybe a microprocessor (or system of interconnected microprocessors), aprogrammable hardware element such as a field-programmable gate array(FPGA), an application specific integrated circuit (ASIC), or anycombination such elements. The processing unit 150 may be configured toperform one or more functions such as image/video construction, systemcontrol, user interface, statistical analysis, and one or moreinferences tasks.

The system 100 (e.g., the processing unit 150) may store the samples{I_(MLS)(k)} in a memory, e.g., a memory resident in the system 100 orin some other system.

In one embodiment, processing unit 150 is configured to operate on thesamples {I_(MLS)(k)} to generate the image or video sequence. In thisembodiment, the processing unit 150 may include a microprocessorconfigured to execute software (i.e., program instructions), especiallysoftware for performing an image/video construction algorithm. In oneembodiment, system 100 is configured to transmit the samples{I_(MLS)(k)} to some other system through a communication channel. (Inembodiments where the spatial patterns are randomly-generated, system100 may also transmit the random seed(s) used to generate the spatialpatterns.) That other system may operate on the samples to construct theimage/video. System 100 may have one or more interfaces configured forsending (and perhaps also receiving) data through one or morecommunication channels, e.g., channels such as wireless channels, wiredchannels, fiber optic channels, acoustic channels, laser-based channels,etc.

In some embodiments, processing unit 150 is configured to use any of avariety of algorithms and/or any of a variety of transformations toperform image/video construction. System 100 may allow a user to choosea desired algorithm and/or a desired transformation for performing theimage/video construction.

In one embodiment, system 100 may include a light transmitter configuredto generate a light beam (e.g., a laser beam), to modulate the lightbeam with a data signal and to transmit the modulated light beam intospace or onto an optical fiber. System 100 may also include a lightreceiver configured to receive a modulated light beam from space or froman optical fiber, and to recover a data stream from the receivedmodulated light beam.

In one embodiment, system 100 may be configured as a low-cost sensorsystem having minimal processing resources, e.g., processing resourcesinsufficient to perform image (or video) construction in user-acceptabletime. In this embodiment, the system 100 may store and/or transmit thesamples {I_(MLS)(k)} so that another agent, more plentifully endowedwith processing resources, may perform the image/video constructionbased on the samples.

In some embodiments, system 100 may include an optical subsystem 105that is configured to modify or condition the light stream L before itarrives at the light modulation unit 110, e.g., as shown in FIG. 2C. Forexample, the optical subsystem 105 may be configured to receive thelight stream L from the environment and to focus the light stream onto amodulating plane of the light modulation unit 110. The optical subsystem105 may include a camera lens (or a set of lenses). The lens (or set oflenses) may be adjustable to accommodate a range of distances toexternal objects being imaged/sensed/captured.

In some embodiments, system 100 may include an optical subsystem 117 todirect the modulated light stream MLS onto a light sensing surface (orsurfaces) of the light sensing device 130.

In some embodiments, the optical subsystem 117 may include one or morelenses, and/or, one or more mirrors.

In some embodiments, the optical subsystem 117 is configured to focusthe modulated light stream onto the light sensing surface (or surfaces).The term “focus” implies an attempt to achieve the condition that rays(photons) diverging from a point on an object plane converge to a point(or an acceptably small spot) on an image plane. The term “focus” alsotypically implies continuity between the object plane point and theimage plane point (or image plane spot)—points close together on theobject plane map respectively to points (or spots) close together on theimage plane. In at least some of the system embodiments that include anarray of light sensing elements, it may be desirable for the modulatedlight stream MLS to be focused onto the light sensing array so thatthere is continuity between points on the light modulation unit LMU andpoints (or spots) on the light sensing array.

In some embodiments, the optical subsystem 117 may be configured todirect the modulated light stream MLS onto the light sensing surface (orsurfaces) of the light sensing device 130 in a non-focusing fashion. Forexample, in a system embodiment that includes only one photodiode, itmay not be so important to achieve the “in focus” condition at the lightsensing surface of the photodiode since positional information ofphotons arriving at that light sensing surface will be immediately lost.

In one embodiment, the optical subsystem 117 may be configured toreceive the modulated light stream and to concentrate the modulatedlight stream into an area (e.g., a small area) on a light sensingsurface of the light sensing device 130. Thus, the diameter of themodulated light stream may be reduced (possibly, radically reduced) inits transit from the optical subsystem 117 to the light sensing surface(or surfaces) of the light sensing device 130. For example, in someembodiments, the diameter may be reduced by a factor of more than 1.5to 1. In other embodiments, the diameter may be reduced by a factor ofmore than 2 to 1. In yet other embodiments, the diameter may be reducedby a factor of more than 10 to 1. In yet other embodiments, the diametermay be reduced by factor of more than 100 to 1. In yet otherembodiments, the diameter may be reduced by factor of more than 400to 1. In one embodiment, the diameter is reduced so that the modulatedlight stream is concentrated onto the light sensing surface of a singlelight sensing element (e.g., a single photodiode).

In some embodiments, this feature of concentrating the modulated lightstream onto the light sensing surface (or surfaces) of the light sensingdevice allows the light sensing device to sense, at any given time, thesum (or surface integral) of the intensities of the modulated lightportions within the modulated light stream. (Each time slice of themodulated light stream comprises a spatial ensemble of modulated lightportions due to the modulation unit's action of applying thecorresponding spatial pattern to the light stream.)

In some embodiments, the modulated light stream MLS may be directed ontothe light sensing surface of the light sensing device 130 withoutconcentration, i.e., without decrease in diameter of the modulated lightstream, e.g., by use of photodiode having a large light sensing surface,large enough to contain the cross section of the modulated light streamwithout the modulated light stream being concentrated.

In some embodiments, the optical subsystem 117 may include one or morelenses. FIG. 2E shows an embodiment where optical subsystem 117 isrealized by a lens 117L, e.g., a biconvex lens or a condenser lens.

In some embodiments, the optical subsystem 117 may include one or moremirrors. In one embodiment, the optical subsystem 117 includes aparabolic mirror (or spherical mirror) to concentrate the modulatedlight stream onto a neighborhood (e.g., a small neighborhood) of theparabolic focal point. In this embodiment, the light sensing surface ofthe light sensing device may be positioned at the focal point.

In some embodiments, system 100 may include an optical mechanism (e.g.,an optical mechanism including one or more prisms and/or one or morediffraction gratings) for splitting or separating the modulated lightstream MLS into two or more separate streams (perhaps numerous streams),where each of the streams is confined to a different wavelength range.The separate streams may each be sensed by a separate light sensingdevice. (In some embodiments, the number of wavelength ranges may be,e.g., greater than 8, or greater than 16, or greater than 64, or greaterthan 256, or greater than 1024.) Furthermore, each separate stream maybe directed (e.g., focused or concentrated) onto the corresponding lightsensing device as described above in connection with optical subsystem117. The samples captured from each light sensing device may be used toconstruct a corresponding image for the corresponding wavelength range.In one embodiment, the modulated light stream is separated into red,green and blue streams to support color (R,G,B) measurements. In anotherembodiment, the modulated light stream may be separated into IR, red,green, blue and UV streams to support five-channel multi-spectralimaging: (IR, R, G, B, UV). In some embodiments, the modulated lightstream may be separated into a number of sub-bands (e.g., adjacentsub-bands) within the IR band to support multi-spectral orhyper-spectral IR imaging. In some embodiments, the number of IRsub-bands may be, e.g., greater than 8, or greater than 16, or greaterthan 64, or greater than 256, or greater than 1024. In some embodiments,the modulated light stream may experience two or more stages of spectralseparation. For example, in a first stage the modulated light stream maybe separated into an IR stream confined to the IR band and one or moreadditional streams confined to other bands. In a second stage, the IRstream may be separated into a number of sub-bands (e.g., numeroussub-bands) (e.g., adjacent sub-bands) within the IR band to supportmultispectral or hyper-spectral IR imaging.

In some embodiments, system 100 may include an optical mechanism (e.g.,a mechanism including one or more beam splitters) for splitting orseparating the modulated light stream MLS into two or more separatestreams, e.g., where each of the streams have the same (or approximatelythe same) spectral characteristics or wavelength range. The separatestreams may then pass through respective bandpass filters to obtaincorresponding modified streams, wherein each modified stream isrestricted to a corresponding band of wavelengths. Each of the modifiedstreams may be sensed by a separate light sensing device. (In someembodiments, the number of wavelength bands may be, e.g., greater than8, or greater than 16, or greater than 64, or greater than 256, orgreater than 1024.) Furthermore, each of the modified streams may bedirected (e.g., focused or concentrated) onto the corresponding lightsensing device as described above in connection with optical subsystem117. The samples captured from each light sensing device may be used toconstruct a corresponding image for the corresponding wavelength band.In one embodiment, the modulated light stream is separated into threestreams which are then filtered, respectively, with a red-pass filter, agreen-pass filter and a blue-pass filter. The resulting red, green andblue streams are then respectively detected by three light sensingdevices to support color (R,G,B) acquisition. In another similarembodiment, five streams are generated, filtered with five respectivefilters, and then measured with five respective light sensing devices tosupport (IR, R, G, B, UV) multi-spectral acquisition. In yet anotherembodiment, the modulated light stream of a given band may be separatedinto a number of (e.g., numerous) sub-bands to support multi-spectral orhyper-spectral imaging.

In some embodiments, system 100 may include an optical mechanism forsplitting or separating the modulated light stream MLS into two or moreseparate streams. The separate streams may be directed to (e.g.,concentrated onto) respective light sensing devices. The light sensingdevices may be configured to be sensitive in different wavelengthranges, e.g., by virtue of their different material properties. Samplescaptured from each light sensing device may be used to construct acorresponding image for the corresponding wavelength range.

In some embodiments, system 100 may include a control unit 120configured to supply the spatial patterns to the light modulation unit110, as shown in FIG. 2F. The control unit may itself generate thepatterns or may receive the patterns from some other agent. The controlunit 120 and the light sensing device 130 may be controlled by a commonclock signal so that the light sensing device 130 can coordinate(synchronize) its action of capturing the samples {I_(MLS)(k)} with thecontrol unit's action of supplying spatial patterns to the lightmodulation unit 110. (System 100 may include clock generationcircuitry.)

In some embodiments, the control unit 120 may supply the spatialpatterns to the light modulation unit in a periodic fashion.

The control unit 120 may be a digital circuit or a combination ofdigital circuits. For example, the control unit may include amicroprocessor (or system of interconnected of microprocessors), aprogrammable hardware element such as a field-programmable programmablegate array (FPGA), an application specific integrated circuit (ASIC), orany combination such elements.

In some embodiments, the control unit 120 may include a random numbergenerator (RNG) or a set of random number generators to generate thespatial patterns or some subset of the spatial patterns.

In some embodiments, system 100 is battery powered. In some embodiments,the system 100 includes a set of one or more solar cells and associatedcircuitry to derive power from sunlight.

In some embodiments, system 100 includes its own light source forilluminating the environment or a target portion of the environment.

In some embodiments, system 100 may include a display (or an interfaceconfigured for coupling to a display) for displaying constructedimages/videos.

In some embodiments, system 100 may include one or more input devices(and/or, one or more interfaces for input devices), e.g., anycombination or subset of the following devices: a set of buttons and/orknobs, a keyboard, a keypad, a mouse, a touch-sensitive sensitive padsuch as a trackpad, a touch-sensitive display screen, one or moremicrophones, one or more temperature sensors, one or more chemicalsensors, one or more pressure sensors, one or more accelerometers, oneor more orientation sensors (e.g., a three-axis gyroscopic sensor), oneor more proximity sensors, one or more antennas, etc.

Regarding the spatial patterns that are used to modulate the lightstream L, it should be understood that there are a wide variety ofpossibilities. In some embodiments, the control unit 120 may beprogrammable so that any desired set of spatial patterns may be used.

In some embodiments, the spatial patterns are binary valued. Such anembodiment may be used, e.g., when the light modulating elements aretwo-state devices. In some embodiments, the spatial patterns are n-statevalued, where each element of each pattern takes one of n states, wheren is an integer greater than two. (Such an embodiment may be used, e.g.,when the light modulating elements are each able to achieve n or moremodulation states). In some embodiments, the spatial patterns are realvalued, e.g., when each of the light modulating elements admits acontinuous range of modulation. (It is noted that even a two-statemodulating element may be made to effectively apply a continuous rangeof modulation by duty cycling the two states during modulationintervals.)

The spatial patterns may belong to a set of measurement vectors that isincoherent with a set of vectors in which the image/video isapproximately sparse (“the sparsity vector set”). (See “Sparse SignalDetection from Incoherent Projections”, Proc. Int. Conf. Acoustics,Speech Signal Processing—ICASSP, May 2006, Duarte et al.) Given two setsof vectors A={a_(i),} and B={b_(i)} in the same N-dimensional space, Aand B are said to be incoherent if their coherence measure μ(A,B) issufficiently small. The coherence measure is defined as:

${\mu( {A,B} )} = {\max\limits_{i,j}{{\langle {a_{i},b_{j}} \rangle }.}}$The number of compressive sensing measurements (i.e., samples of thesequence {I_(MLS)(k)} needed to construct an N-pixel image (or N-voxelvideo sequence) that accurately represents the scene being captured is astrictly increasing function of the coherence between the measurementvector set and the sparsity vector set. Thus, better compression can beachieved with smaller values of the coherence.

In some embodiments, the measurement vector set may be based on a code.Any of various codes from information theory may be used, e.g., codessuch as exponentiated Kerdock codes, exponentiated Delsarte-Goethalscodes, run-length limited codes, LDPC codes, Reed Solomon codes and ReedMuller codes.

In some embodiments, the measurement vector set corresponds to apermuted basis such as a permuted DCT basis or a permuted Walsh-Hadamardbasis, etc.

In some embodiments, the spatial patterns may be random or pseudo-randompatterns, e.g., generated according to a random number generation (RNG)algorithm using one or more seeds. In some embodiments, the elements ofeach pattern are generated by a series of Bernoulli trials, where eachtrial has a probability p of giving the value one and probability 1−p ofgiving the value zero. (For example, in one embodiment p=1/2.) In someembodiments, the elements of each pattern are generated by a series ofdraws from a Gaussian random variable.)

The system 100 may be configured to operate in a compressive fashion,where the number of the samples {I_(MLS)(k)} captured by the system 100is less than (e.g., much less than) the number of pixels in the image(or video) to be constructed from the samples. In many applications,this compressive realization is very desirable because it saves on powerconsumption, memory utilization and transmission bandwidth consumption.However, non-compressive realizations are contemplated as well.

In some embodiments, the system 100 is configured as a camera or imagerthat captures information representing an image (or a series of images)from the external environment, e.g., an image (or a series of images) ofsome external object or scene. The camera system may take differentforms in different application domains, e.g., domains such as visiblelight photography, infrared photography, ultraviolet photography,high-speed photography, low-light photography, underwater photography,multi-spectral imaging, hyper-spectral imaging, etc. In someembodiments, system 100 is configured to operate in conjunction with (oras part of) another system, e.g., in conjunction with a microscope, atelescope, a robot, a security system, a surveillance system, a firesensor, a node in a distributed sensor network, etc.

In some embodiments, system 100 is configured as a spectrometer.

In some embodiments, system 100 may also be configured to operate as aprojector. Thus, system 100 may include a light source, e.g., a lightsource located at or near a focal point of optical subsystem 117. Inprojection mode, the light modulation unit 110 may be supplied with animage (or a video sequence) so that the image (or video sequence) can bedisplayed on a display surface (e.g., screen).

In one realization 200 of system 100, the light modulation unit 110 maybe realized by a plurality of mirrors, e.g., as shown in FIG. 3A. (Themirrors are collectively indicated by the label 110M.) The mirrors 110Mare configured to receive corresponding portions of the light L receivedfrom the environment, albeit not necessarily directly from theenvironment. (There may be one or more optical elements, e.g., one ormore lenses along the input path to the mirrors 110M.) Each of themirrors is configured to controllably switch between two orientationstates. In addition, each of the mirrors is configured to (a) reflectthe corresponding portion of the light onto a sensing path 115 when themirror is in a first of the two orientation states and (b) reflect thecorresponding portion of the light away from the sensing path when themirror is in a second of the two orientation states.

In some embodiments, the mirrors 110M are arranged in an array, e.g., atwo-dimensional array or a one-dimensional array. Any of various arraygeometries are contemplated. For example, in different embodiments, thearray may be a square array, a rectangular array, a hexagonal array,etc. In some embodiments, the mirrors are arranged in a spatially-randomfashion.

The mirrors 110M may be part of a digital micromirror device (DMD). Forexample, in some embodiments, one of the DMDs manufactured by TexasInstruments may be used.

The control unit 120 may be configured to drive the orientation statesof the mirrors through the series of spatial patterns, where each of thepatterns of the series specifies an orientation state for each of themirrors.

The light sensing device 130 may be configured to receive the lightportions reflected at any given time onto the sensing path 115 by thesubset of mirrors in the first orientation state and to generate ananalog electrical signal I_(MLS)(t) representing a cumulative intensityof the received light portions as function of time. As the mirrors aredriven through the series of spatial patterns, the subset of mirrors inthe first orientation state will vary from one spatial pattern to thenext. Thus, the cumulative intensity of light portions reflected ontothe sensing path 115 and arriving at the light sensing device will varyas a function time. Note that the term “cumulative” is meant to suggesta summation (spatial integration) over the light portions arriving atthe light sensing device at any given time. This summation may beimplemented, at least in part, optically (e.g., by means of a lensand/or mirror that concentrates or focuses the light portions onto aconcentrated area as described above).

System realization 200 may include any subset of the features,embodiments and elements discussed above with respect to system 100. Forexample, system realization 200 may include the optical subsystem 105 tooperate on the incoming light L before it arrives at the mirrors 110M,e.g., as shown in FIG. 3B.

In some embodiments, system realization 200 may include the opticalsubsystem 117 along the sensing path as shown in FIG. 4. The opticalsubsystem 117 receives the light portions reflected onto the sensingpath 115 and directs (e.g., focuses or concentrates) the received lightportions onto a light sensing surface (or surfaces) of the light sensingdevice 130. In one embodiment, the optical subsystem 117 may include alens 117L, e.g., as shown in FIG. 5A.

In some embodiments, the optical subsystem 117 may include one or moremirrors, e.g., a mirror 117M as shown in FIG. 5B. Thus, the sensing pathmay be a bent path having more than one segment. FIG. 5B also shows onepossible embodiment of optical subsystem 105, as a lens 105L.

In some embodiments, there may be one or more optical elementsintervening between the optical subsystem 105 and the mirrors 110M. Forexample, as shown in FIG. 5C, a TIR prism pair 107 may be positionedbetween the optical subsystem 105 and the mirrors 110M. (TIR is anacronym for “total internal reflection”.) Light from optical subsystem105 is transmitted through the TIR prism pair and then interacts withthe mirrors 110M. After having interacted with the mirrors 110M, lightportions from mirrors in the first orientation state are reflected by asecond prism of the pair onto the sensing path 115. Light portions frommirrors in the second orientation state may be reflected away from thesensing path.

Delivering Spatial Subsets of Modulated Light Stream to Two or MoreLight Sensing Elements, to Decrease Image Acquisition Time

In one set of embodiments, a system 600 for operating on light may beconfigured as shown in FIG. 6. System 600 includes the light modulationunit 110 as described above, and also includes an optical subsystem 620,a plurality of light sensing devices (collectively referenced by itemnumber 630), and a sampling subsystem 640. (Furthermore, any subset ofthe features, embodiments and elements discussed above with respect tosystem 100 and system realization 200 may be incorporated into system600.)

As described above, the light modulation unit 110 is configured tomodulate an incident stream of light L with a sequence of spatialpatterns in order to produce a modulated light stream MLS. The lightmodulation unit includes an array of light modulating elements. Thelight modulation unit 110 may be realized in any of the various waysdescribed above in connection with system 100 and system realization200. (In particular, the fact that FIG. 6 shows light entering at theleft side of the LMU and exiting the right side is not meant to limitthe LMU to transmissive embodiments.)

The optical subsystem 620 is configured to deliver light from each of aplurality of spatial subsets of the modulated light stream onto arespective one of the light sensing devices 630. (The linear arrangementof the light sensing devices in FIG. 6 is not meant to be limiting.Various other arrangements, especially two-dimensional arrangements, arepossible.) (The fact that the light sensing devices in FIG. 6 are shownas being separated from each other is not meant to be limiting. In someembodiments, the light sensing devices are physically adjacent (orcontiguous), arranged in an array, or in close proximity to oneanother.) In other words, light from a first subset of the modulatedlight stream is delivered onto a first light sensing device; light froma second subset of the modulated light stream is delivered onto a secondlight sensing device; and so on. The spatial subsets of the modulatedlight stream are produced by respective subsets of the array of lightmodulating elements. Each subset of the array of light modulatingelements produces the respective spatial subset of the modulated lightstream by modulating a respective spatial subset of the incident lightstream. The light sensing devices 630 are configured to generaterespective electrical signals. Each of the electrical signals representsintensity of the respective spatial subset of the modulated light streamas a function of time.

By saying that light from a given spatial subset of the modulated lightstream is “delivered” onto the respective light sensing device meansthat most of the flux (i.e., photons) arriving at the respective lightsensing device comes from the spatial subset. However, it should beunderstood that the systems and methods described herein tolerate asmall amount of light crossover in the delivery of light from spatialsubsets of the modulated light stream onto the respective light sensingdevices. Light crossover occurs when a photon of a given spatial subsetarrives a light sensing device other than the respective light sensingdevice (e.g., when a photon from the first spatial subset arrives at thesecond light sensing device). Light crossover might occur, e.g., forphotons produced from the light modulation unit along the boundaries ofthe spatial subsets, e.g., due to imperfections in the optical subsystem620. Crossover for a given light sensing device might be defined as theratio of the amount of light power received at the light sensing devicefrom spatial subsets other than the respective spatial subset to theamount of light power received at the light sensing device from therespective spatial subset. The crossover for a given system might bedefined, e.g., as the maximum crossover taken over all the light sensingdevices.

In different embodiments, different amounts of crossover may betolerated, e.g., depending on the targeted quality of image/videoreconstruction. For example, in different sets of embodiments, themaximum crossover may be required, respectively, to be less than ½, lessthan ¼, less than ⅛, less than 1/32, less than 1/128, less than 1/1024,less than 1/4096, less than 1/16384.

Furthermore, the “delivery” of light from a given spatial subset of themodulated light stream onto the respective light sensing device does notrequire that 100% of the photons from the spatial subset arrive at therespective light sensing device (or that 100% of the photons that arrivebe converted into electrical charge). Small amounts of light loss may beunavoidable. It is perhaps preferable that the percentage of light lossbe uniform from one spatial subset to the next. In some embodiments, thepercentage of light loss may be determined per spatial subset bycalibration, and then, used to compensate the measurements obtained fromeach light sensing device, in an attempt to obtain compensated valuesthat approximate the measurements that would have been obtained from aperfectly uniform system. In some embodiments, or, under strong ambientlight conditions, significant amounts of light loss may be tolerated.

In some embodiments, the spatial subsets of the modulated light streamare defined in terms of the light sensing device that detects thelargest percentage of the photons traversing a given point on a givencross section of the modulated light stream. For example, taking a crosssection of the modulated light stream at or near the plane of the lightmodulating array, a point in that cross section may be assigned to thek^(th) spatial subset if the k^(th) light sensing device detects more ofthe photons that traverse that point than any other one of the lightsensing devices. The subsets of the array of light modulating elementsand the subsets of the incident light stream may be similarly defined.

In some embodiments, the spatial subsets of the modulated light streammay be non-overlapping subsets when viewed in a given cross section.Furthermore, the spatial subsets may cover the extent of the crosssection. Likewise, the subsets of the array of the light modulatingelements may be non-overlapping subsets and cover the entirety of thearray.

In some embodiments, the spatial subsets of the modulated light streammay be uniform in size and shape. (Likewise, the subsets of the array oflight modulating elements may be uniform in size and shape.) However, inother embodiments, the spatial subsets (and the array subsets) may havesizes and/or shapes that are non-uniform. For example, in oneembodiment, subsets at or near the center of the array of lightmodulating elements may be smaller in diameter (or area), and thus,packed more densely, than subsets at or near the boundary of the array.

In some embodiments, the spatial subsets of the modulated light streamand the subsets of the array of light modulating elements are simplyconnected regions. A region is said to be “simply connected” when anytwo points in the region are connected by a path that lies entirelywithin the region, and, any such path between two points in the regioncan be continuously deformed, staying within the region, into any othersuch path between the two points. As a consequence, a simply connectedregion doesn't have any holes and doesn't include separated pieces.

In some embodiments, the spatial subsets of the modulated light streamand the subsets of the array of light modulating elements are convexregions. In one embodiment, the spatial subsets and the array subsetsare rectangles (e.g., squares). In another embodiment, at least some ofthe subsets are hexagons.

In some embodiments, the subsets of the array of light modulatingelements are defined by a Voronoi tessellation of the array based on adiscrete set of points in the plane of the array, where the set ofpoints includes one point for each of the light sensing devices.

The sampling subsystem 640 is configured to acquire samples of theelectrical signals generated respectively by the light sensing devices630. The samples include sample subsets that correspond respectively tothe electrical signals. Each sample subset includes a plurality ofsamples of the respective electrical signal. In the discussion below,the electrical signals are denoted as I₁(t), I₂(t), . . . , I_(L)(t),where L is the number of the light sensing devices, and the samplesubsets are denoted as {I₁(k)}, {I₂(k)}, . . . , {I_(L)(k)}. The numberL is greater than one but less than N, where N is the number of lightmodulating elements in the array of light modulating elements.

In various embodiments, the number N may take a wide variety of values.For example, in different sets of embodiments, N may be, respectively,in the range [64, 256], in the range [256, 1024], in the range[1024,4096], in the range [2¹²,2¹⁴], in the range [2¹⁴,2¹⁶], in therange [2¹⁶,2¹⁸], in the range [2¹⁸,2²⁰], in the range [2²⁰,2²²], in therange [2²²,2²⁴], in the range [2²⁴,2²⁶], in the range from 2²⁶ toinfinity. The particular value used in any given embodiment may dependon one or more factors specific to the embodiment.

Likewise, the number L may take any of a wide variety of values. Forexample, in different sets of embodiments, L may be, respectively, inthe range [2,4], in the range [4,8], in the range [8,16], in the range[16,32], in the range [32,64], in the range [64, 256], in the range[256,1024], in the range [1024,4096], in the range [2¹²,2¹⁴], in therange [2^(14,)2¹⁶], in the range [2¹⁶,2¹⁸], in the range [2¹⁸,2²⁰], inthe range [2²⁰,2²²], in the range [2²²,2²⁴], in the range [2²⁴,2²⁶].

Furthermore, the ratio N/L make take any of a wide of values, e.g., anyvalue in the range from 2 to N/2. In different sets of embodiments, theratio N/L may be, respectively, in the range [2,4], in the range [4,8],in the range [8,16], in the range [16,64], in the range [64,256], in therange [256,1024], in the range [1024,4096], in the range [4096,2¹⁴], inthe range [2¹⁴,2¹⁶], in the range [2¹⁶,2¹⁸], in the range [2²⁰,2²²], inthe range [2²²,2²⁴], in the range from 2²⁴ to infinity.

As noted above, the samples acquired by the sampling subsystem 640include a sample subset for each of the electrical signals. In someembodiments, each sample subset is acquired over the same window intime, or, approximately the same window in time.

In some embodiments, the samples of each subset are acquired at the samerate. In other embodiments, not all the sample subsets are acquired atthe same rate.

In some embodiments, each of the sample subsets is acquired at the samerate, and, the samples I₁(k), I₂(k), . . . , I_(L)(k) corresponding tosame time index value k are acquired at the same time t_(k), or atleast, close together in time. The samples I₁(k), I₂(k), . . . ,I_(L)(k) corresponding to the same time index value k may be referred toherein as a “sample group”.

In some embodiments, each sample group {I₁(k), I₂(k), . . . , I_(L)(k)}corresponds to a respective one of the patterns of the sequence ofspatial patterns. In particular, the sample I_(j)(k), j=1, 2, 3, . . . ,L, may represent an inner product between (a) the restriction of atime-slice of the incident light stream to the j^(th) subset of thearray of light modulating elements and (b) the restriction of a k^(th)one of the spatial patterns to the j^(th) subset.

Each of the L sample subsets is usable to construct a respectivesub-image of an image, as suggested in FIG. 7. (See also FIG. 14.) (CAis an acronym for “construction algorithm”.) In other words, the samplesubset {I₁(k)} is usable to construct a first sub-image; the samplesubset {I₂(k)} is usable to construct a second sub-image; and so on.Each sub-image represents a respective one of the spatial subsets of theincident light stream (over the time interval of acquisition). Thesub-images may be joined together to form the image. The image mayrepresent the whole of the incident light stream (or the portion of theincident light stream that impinges upon the array of light modulatingelements) over the time interval of acquisition. The number of pixels ineach sub-image may equal the number of light modulating elements in therespective subset of the array of light modulating elements. Each pixelof the j^(th) sub-image may correspond to a respective one of the lightmodulating elements in the j^(th) subset of the array. Thus, the finalimage may be in an N-pixel image, with each of the N pixelscorresponding to a respective one of the N light modulating elements ofthe array. In some embodiments, each sub-image is N/L pixels in size.However, in other embodiments, the sub-images may be non-identicallysized.

Each sub-image may be constructed using the respective sample subset andthe restriction of the sequence of spatial patterns to the correspondingsubset of the array of light modulating elements. For example, toconstruct the first sub-image, the construction algorithm operates onthe first sample subset (obtained from the first light sensing device)and the spatial patterns restricted to the first subset of the array oflight modulating elements. The construction of the sub-images may beperformed by system 600 and/or by some other system. In the latter case,system 600 may send the sample subsets to the other system. Thesub-images may be constructed using any image construction algorithmknown in the field of compressive sensing, or any combination of suchalgorithms.

In some embodiments, each sample subset is a compressed representationof the corresponding sub-image, i.e., the number of samples m_(SS) inthe sample subset is smaller than the number of pixels n_(PSI) in thecorresponding sub-image. For example, in different embodiments, theratio of m_(SS)/n_(PSI) may be, respectively, less than or equal to 0.8,less than or equal to 0.6 , less than or equal to 0.4, less than orequal to 0.3, less than or equal to 0.2, less than or equal to 0.1, lessthan or equal to 0.05, less than or equal to 0.025. In some embodiments,no compression is achieved, i.e., m_(SS) is greater than or equal ton_(PSI). (Such might be the case when the ratio N/L is small or when thesignal-to -noise ratio is small.)

Suppose that the L light sensing devices receive light, respectively,from L equal-sized non-overlapping subsets of the array of N lightmodulating elements. (See FIG. 14 for an example of the L=4 case.) Eachof the L sample subsets is then usable to construct an N/L pixelsub-image of an N-pixel image. The L sample subsets are referred tocollectively as an encoded data representation of the N-pixel image (orsimply as an “encoded image”) because they are usable to produce theN-pixel image by the above-described process. The number of samplesm_(SS) in each sample subset is selected to be greater than or equal tothe minimum number m_(N/L) of compressive-sensing measurements requiredto reconstruct an N/L-pixel image with given accuracy. (Thesecompressive-sensing measurements are inner products between theN/L-pixel image and N/L-element measurement vectors.) If the samplegroups are acquired at a rate of R_(G) groups per unit time, and eachsample group corresponds to a respective one of the spatial patterns,the time required to obtain the encoded data representation of theN-pixel image is m_(SS)/R_(G). Because the minimum number m_(N/L)decreases as L increases, a system designer may decrease the timerequired to acquire the encoded data representation by increasing L. Inparticular, the designer may select L and m_(SS) so that m_(SS)≧m_(N/L)and so that m_(SS)/R_(G) is less than or equal to a target time ofacquisition for the encoded image. In practice, one does not need toknow the minimum m_(N/L) exactly. It suffices if one has an upper boundU^(N/L)≧m_(N/L)(e.g., an upper bound determined empirically, or based ontheoretical considerations, or based on a worst case assumption such asN/L≧m_(N/L)) and the upper bound decreases significantly with increasingL. In that case, the value of L may be selected so that m_(SS)=U_(N/L)and U_(N/L)/R_(G) is less than or equal to the target acquisition time.In one embodiment, U_(N/L) is defined by U_(N/L)=U_(N)/L, where U_(N) isan upper bound on the minimum number of compressive sensing measurementsrequired to reconstruct an N-pixel image.

As noted above, the final image may be an N-pixel image. More generally,the final image may be an n-pixel image, where n is less than or equalto N. The spatial patterns may be designed to support a value of n lessthan N, e.g., by forcing the array of light modulating elements tooperate at a lower effective resolution than the physical resolution N.For example, the spatial patterns may be designed to force each 2×2 cellof light modulating elements to act in unison. At any given time, themodulation state of the four elements in a 2×2 cell will agree. Thus,the effective resolution of the array of light modulating elements isreduced to N/4. This principle generalizes to any cell size, to cells ofany shape, and to collections of cells with non-uniform cell size and/orcell shape. For example, a collection of cells of size k_(H)×k_(V),where k_(H) and k_(V) are positive integers, would give an effectiveresolution equal to N/(k_(H)k_(V)). In one alternative embodiment, cellsnear the center of the array may have smaller sizes than cells near theperiphery of the array.

Assuming that the effective resolution of the array of light modulatingelements is n, with n≦N, that the above-described cells areuniformly-sized, and that each light modulating element “sees” an equalportion of the array of light modulating elements, then each subset ofthe array includes n/L effective elements, and each of the L samplesubsets is usable to construct a respective n/L-pixel sub-image. The Lsub-images may then be stitched together to form the n-pixel image.Thus, the L sample subsets are referred to collectively as an encodedrepresentation of the n-pixel image (or simply as an “encoded image”).The number of samples m_(SS) in each sample subset may be set to a valuegreater than or equal to the minimum number m_(n/L) ofcompressive-sensing samples required to reconstruct an n/L-pixel image.If the sample groups are acquired at a rate of R_(G) groups per unittime, and each sample group corresponds to a respective one of thespatial patterns, the time required to acquire the encodedrepresentation of the n-pixel image is m_(SS)/R_(G). Because m_(n/L)decreases as n/L decreases, m_(n/L) may be decreased by increasing Land/or by decreasing n. Thus, a system designer may select L and m_(SS)so that m_(SS)≧m_(n/L) and so that m_(SS)/R_(G) is less than or equal toa target time of acquisition for the encoded image. Furthermore, system600 may adjust (e.g., dynamically adjust) the acquisition time for anencoded image (or the frame rate for a sequence of encoded images, inthe case of video) by adjusting the value of n. The value of n may beadjusted by changing the sequence of spatial patterns being supplied tothe light modulation unit 110.

In some embodiments, the sampling subsystem 640 may oversample theelectrical signals relative to the rate at which the spatial patternsare applied by the light modulation unit 110. Oversampling meansacquiring two or more sample groups per spatial pattern. In other words,for each electrical signal, the sampling subsystem may acquire more thanone sample per spatial pattern. For each electrical signal, the two ormore samples per spatial pattern may be combined (e.g., by averaging) toobtain a single refined sample per spatial pattern. This combining maybe performed prior to construction of the sub-images. Sampling subsystem640 may be configured to perform this combining The constructionalgorithm preferably uses the refined samples. Thus, with anoversampling factor of c_(OS>)1, the sampling subsystem 640 may gatherc_(OS)m_(SS) samples of each electrical signal. Then the c_(OS)m_(SS)samples may be processed to obtain m_(SS) refined samples. The m_(SS)refined samples of each electrical signal may be used to construct thecorresponding sub-image.

As described above, the light modulation unit 110 modulates the incidentlight stream with a sequence of the spatial patterns. Let R_(P) denotethe maximum rate (in spatial patterns per unit time) that the lightmodulation unit 110 can attain. In some embodiments, the rate R_(P) maybe slower than the rate R_(G) at which sample groups are acquired, i.e.,the rate at which each electrical signal is sampled. (For example, thiscondition may hold for a system that includes a separate ADC per lightsensing device, or for a system that has a small number of light sensingdevices per ADC.) Thus, the rate R_(P) may be the limiting factor whenattempting to meet a desired acquisition time for an encodedrepresentation of an n-pixel image, n≦N. However, recall from above thatthe minimum number m_(n/L) of compressive-sensing samples required toreconstruct an n/L-pixel image with given accuracy decreases as Lincreases (and as n decreases). Thus, the number m_(SS) of samples ineach sample subset and the number L of light sensing devices may beselected so that m_(SS) is greater than or equal to m_(n/L) andm_(SS)/R_(P) is less than or equal to a target acquisition time, orequivalently, so that R_(P)/m_(SS) is greater than or equal to a targetframe rate.

As described above, the sampling subsystem 640 acquires samples of theelectrical signals, where the samples include L sample subsets acquiredrespectively from the L light sensing device 630. Each sample subset isusable to construct a respective n/L-pixel sub-image, where n≦N. Thesampling subsystem 640 may be configured so that the number m_(SS) ofsamples per sample subset is a programmable parameter. (In someembodiments, the number m_(SS) is dynamically programmable, i.e., can bechanged while the system 600 is acquiring the encoded images, e.g.,between one encoded image and the next. This feature allows the system600 to achieve different tradeoffs between image quality and acquisitiontime for the samples. Larger values of m_(SS) imply higher image qualitybut longer acquisition time (or lower frame rate). Conversely, smallervalues of m_(SS) imply lower image quality but shorter acquisition time(or higher frame rate). The system 600 (e.g., processing unit 150) mayprovide a user interface through which a user may control the tradeoffbetween image quality and acquisition time.

In some embodiments, system 600 may include the processing unit 150described above, e.g., as shown in FIG. 8. The processing unit may beconfigured to operate on each sample subset to construct the respectivesub-image using any of the image construction algorithms known in thefield of compressive sensing. (The processing unit may also join thesub-images to form the final image.)

In some embodiments, system 600 may transmit the sample subsets to aremote system through a communication channel, e.g., as variouslydescribed above. The remote system may then construct the sub-imagesfrom the sample subsets. In some embodiments, system 600 is configuredfor low power consumption, and thus, may not be capable of constructingthe sub-images in user-acceptable time. (The construction algorithms aretypically computationally intensive.)

In some embodiments, the sampling subsystem 640 may be configured toacquire the samples of the electrical signals in groups, where each ofthe groups includes at least one (or exactly one) sample of each of theelectrical signals. Each of the groups may correspond to a respectiveone of the spatial patterns, i.e., may be captured in response to thelight modulation unit's assertion of a respective one of the spatialpatterns. The spatial pattern may be held constant during theacquisition of the group.

The sampling subsystem 640 may acquire the groups of samples at a grouprate of R_(G) groups per unit time. In some embodiments, the maximumgroup rate R_(G) may be lower than the maximum pattern modulation rateR_(P). (For example, this rate condition may hold for system embodimentswhere the number L of light sensing devices is significantly larger thanthe number of ADCs.) Thus, the acquisition time for an encoded image maybe determined by the ratio m_(SS)/R_(G), where m_(SS) is the number ofgroups acquired for the encoded image (or equivalently, the number ofsamples per sample subset). However, recall that the minimum numberm_(n/L) of compressive sensing samples required to reconstruct ann/L-pixel image with given accuracy decreases as L increases (and as ndecreases). Thus, a system designer may select the value of Lsufficiently large so that m_(SS)≧m_(n/L), and, m_(SS)/R_(G) is lessthan or equal to a target acquisition time, or, equivalently,R_(G)/m_(SS) is greater than or equal to a target frame rate.

In some embodiments, the sampling subsystem 640 may include a pluralityof analog-to-digital converters (ADCs), e.g., as shown in FIG. 9. Eachof the ADCs is configured to acquire the samples of the electricalsignal generated by a respective one of the light sensing devices. Inother words, each ADC acquires a corresponding one of the sample subsets{I₁(k)}, {I₂(k)}, . . . , {I_(L)(k)}.

In some embodiments, the ADCs may be configured to operate in parallel,e.g., by virtue of being clocked by the same sample conversion clock.

In some embodiments, each ADC includes its own preamplifier, e.g., apreamplifier with controllable gain.

In some embodiments, the ADCs acquire samples in synchronization withthe application of spatial patterns.

In some embodiments, the ADCs are configured so they each have the samebit resolution, i.e., the number of bits per sample from each ADC is thesame. However, in other embodiments, some of the ADCs may have differentbit resolutions by design.

In some embodiments, the optical subsystem 620 includes a plurality oflenses, e.g., as shown in FIG. 10. Each of the lenses is configured todirect the light from a respective one of the spatial subsets of themodulated light stream onto a light sensing surface of a respective oneof the light sensing devices 630. The one-dimensional arrangement of thelenses shown in FIG. 10 is not meant to be limiting. Various otherarrangements, especially two-dimensional arrangements, are alsocontemplated. For example, in the embodiment of FIG. 11, the opticalsubsystem 620 includes a 2×2 lenslet array 620LA that directs light ontoa 2×2 array 630Q of light sensing elements. (The 2×2 array of lightsensing elements may be integrated into a quadrant photodiode.) Eachlenslet of the lenslet array directs (e.g., concentrates) light from acorresponding quadrant of the modulated light stream MLS onto acorresponding one of the light sensing elements, e.g., onto acorresponding quadrant of a quadrant photodiode. More generally, thelenslet array may have any desired number of lenslets and any desiredgeometric configuration. In some embodiments, the array of light sensingelements may agree in number and geometric configuration with thelenslet array. For example, if the lenslet array is an L_(H)×L_(V)rectangular array, with L_(H)and L_(V) being integers greater than one,the array of light sensing elements may also be an L_(H)×L_(V)rectangular array. As another example, if the lenslet array is ahexagonal array, the light sensing array may also be hexagonal. However,in other embodiments, the array of light sensing elements and thelenslet array do not agree in number or geometry. For example, in oneembodiment, each lenslet may direct light onto a corresponding group oftwo or more light sensing elements.

FIG. 12 shows an embodiment of system 600 where the light modulationunit 110 is realized by the array of mirrors 110M (e.g., a DMD), and theoptical subsystem 620 includes image relay optics 620RO and the lensletarray 620LA. The lenslet array 620LA directs spatial subsets of themodulated light stream onto respective light sensing elements of thesensor array 630SA. (In one embodiment, the sensor array 630SA is aquadrant photodiode. However, a wide vareity of other embodiments arecontemplated.) The electrical signals generated by the sensor array630SA are sampled by the sampling subsystem 640. Note also that FIG. 12includes an optical subsystem 1210 and a TIR prism pair 1220. (Theoptical subsystem 1210 focuses the incident light stream L onto thearray of mirrors 110M. The optical subsystem may be a lens, e.g., acamera lens.) (See the above discussion regarding the TIR prism pair 107of FIG. 5C.) As described above, the lenslet array 620LA may have anydesired number of lenslets and any desired geometric arrangement oflenslets.

FIG. 13 shows example waveforms for the electrical signals I₁(t), I₂(t),I₃(t) and I₄(t) in a case where the light modulation unit 110 is a DMD.These example waveforms are not meant to limit the scope of meaning ofthe term “electrical signal” as used herein. In other embodiments, thewaveforms may have different features, e.g., different transientbehavior upon transitions between spatial patterns.

FIG. 14 illustrates one embodiment of the processes of sampleacquisition and image construction in a case where there are four lightsensing devices LSD1-LSD4. At the left is a visualization of themodulated light stream at the modulation plane of the light modulationunit 110 (at a snapshot in time). The light modulation unit includes Nlight modulating elements. The four light sensing devices receive lightfrom four respective quadrants of the modulation plane. The electricalsignal generated by each light sensing device is sampled to generate arespective sample subset. (See the A/D convert blocks.) Each samplesubset is processed by a construction algorithm (CA) to generate arespective sub-image. (The construction algorithm may be anyconstruction algorithm known in the field of compressive sensing.) Thesub-images are stitched together to form an N-pixel image.

In some embodiments, the optical subsystem 620 may include a fibercoupling device 620FC and a plurality of optical fiber bundles 620FB,e.g., as shown in FIG. 15. The fiber coupling device is configured tocouple light from each of the spatial subsets of the modulated lightstream onto a respective one of the fiber bundles. Each fiber bundle isconfigured to deliver the light from the respective spatial subset ontoa light sensing surface of a respective one of the light sensing devices630. Because light is delivered by means of optical fiber bundles, thelight sensing devices 630 need not be physically adjacent or locatednear each other. A system designer may position the light sensingdevices at any set of locations that are convenient, and then, selectappropriate lengths for the optical fiber bundles so that they willreach those locations.

In some embodiments, each fiber bundle is a single optical fiber.

In some embodiments, the fiber coupling device includes a set of L fibertapers. Each fiber taper is configured to conduct light from arespective one of the spatial subsets of the modulated light stream ontoa respective one of the fiber bundles (or optical fibers).

FIG. 16 shows an embodiment of system 600 where the light modulationunit 110 is realized by the array of mirrors 110M (e.g., a DMD), and theoptical subsystem 620 includes image relay optics 630RO, the fibercoupling device 620FC and the fiber bundles 620FB.

In some embodiments, the optical subsystem 620 includes a plurality oflight pipes 620P (e.g., hollow tubes or tunnels or guides) whoseinternal surfaces are reflective, e.g., as suggested in FIG. 17A. Eachlight pipe has two open ends. A first end of each pipe couples (or, isin close proximity) to a corresponding subset of the array of lightmodulating elements so that modulated light the array subset enters thepipe. A second end of each pipe couples (or, is in close proximity) tothe light sensing surface of a corresponding light sensing device sothat modulated light from the corresponding array subset is delivered tothat light sensing surface. Each light pipe serves as a waveguide toguide the light from the corresponding spatial subset to thecorresponding light sensing device. The internal surfaces of the pipesmay be polished.

FIG. 17B shows an embodiment of the light pipes 620P configured to guidelight from four quadrants of the array of light modulating elements ontofour respective light sensing devices. In other embodiments, the lightpipes 620P may include desired any number of pipes in any desiredgeometric arrangement.

It should be understood that the term “pipe”, as used herein, includeswithin its scope of meaning the possibility that the cross section ordiameter of a pipe may vary along its length.

In one set of embodiments, the light sensing devices are arranged in anarray (“the light sensing array”).

The light sensing array may be smaller in area (or diameter) than thearray of light modulating elements. Thus, in some embodiments, theoptical subsystem 620 may be configured to concentrate the modulatedlight stream onto the light sensing array. See, e.g., FIGS. 18 and 19.The diameter (or the cross sectional area) of the modulated light beamdecreases as it moves from the array of light modulating elements to thelight sensing array. (The optical subsystem 620 may include one or morelenses and/or one or more mirrors to achieve the concentrating action.)The optical subsystem may also be configured so that the modulated lightstream comes into focus at the light sensing array.

Alternatively, the light sensing array may be of the same size as thearray of light modulating elements. Thus, in some embodiments, theoptical subsystem 620 may be configured to maintain the size of themodulated light stream as it moves from the array of light modulatingelements to the light sensing array. The optical subsystem may also beconfigured so that the modulated light stream comes into focus at thelight sensing array.

As yet another alternative, the light sensing array may be larger insize than the array of light modulating elements. Thus, the opticalsubsystem 620 may be configured to increase the size of the modulatedlight stream as it moves from the array of light modulating elements tothe light sensing array. The optical subsystem may also be configured sothat the modulated light stream comes into focus at the light sensingarray.

In some embodiments, the optical subsystem 620 includes a focusing lens620FL, e.g., as shown in FIG. 18. The light sensing devices 630 may belocated at an image plane of the focusing lens. Furthermore, an objectplane of the focusing lens may coincide with a modulating plane of thelight modulation unit 110. The focusing lens focuses the modulated lightstream onto the light sensing devices. Each light sensing devicereceives light from a corresponding spatial portion of the array oflight modulating elements. FIG. 19 shows an embodiment where the lightsensing devices 630 are arranged in an 8×8 array 630A (e.g., an arrayfabricated on an integrated circuit). The samples obtained from eachlight sensing device may be used to construct 1/64^(th) of the finalimage. While FIG. 19 shows an 8×8 array of light sensing devices, inother embodiments, the array may have any desired dimensions and anydesired geometric configuration. In an embodiment that uses anL_(H)×L_(V) array, where L_(H) and L_(V) are positive integers, thesamples obtained from each array element are used to construct onesub-image among L_(H)*L_(V) sub-images. The L_(H)*L_(V) sub-images arestitched together to form the final image. The product L=L_(H)*L_(V) maybe less than the number N of light modulating elements in the lightmodulation unit 110.

In some embodiments, the light sensing devices 630 may be configured sothat they each have the same sensitivity to light. However, in otherembodiments, the light sensing devices may have differing lightsensitivities by design.

As described above, the light sensing devices 630 may be arranged in arectangular array. In some embodiments, the sampling subsystem 640 mayinclude one ADC per column of the array, e.g., as shown in FIG. 20.Thus, a whole row of sensor values may be read from the array inparallel. (The blocks indicated by integer pairs i,j are the lightsensing devices.) (The term “sensor” is used here as a synonym for“light sensing device”.) The row decoders 2010 are configured to selectone of the rows in response to received row address data. The lightsensing devices of the selected row then output their voltage signals(or current signals or charge signals) respectively to column amplifiersG1-G4. The converters ADC#1-ADC#4 then digitize (sample) the respectiveamplified signals in parallel. A controlling agent (e.g., processingunit 150) may drive the row address so that the rows are read out in acyclic fashion. While FIG. 20 shows a 4×4 array of light sensingdevices, the described principle generalizes to arrays having any numberof rows and any number of columns.

In some embodiments, the whole array of light sensing devices share asingle ADC, e.g., as suggested in FIG. 21. The light sensing devices ofa selected row transfer their voltage signals (or charge signals)respectively to buffers Buf#1-Buf#4. Multiplexer 2110 transfers aselected one of the buffered signals to its output based on the columnaddress. The selected signal is then amplified by amplifier 2120 andthen digitized by ADC 2130. A controlling agent (e.g., processing unit150) may drive the column address so that the buffered signals of theselected row are digitized sequentially. Furthermore, the controllingagent may drive the row address and column address so that the entirearray is digitized, e.g., in a raster fashion. While FIG. 21 shows a 4x4array of light sensing devices, the described principle generalizes toarrays of any size. In some embodiments, the controlling agent mayimplement windowing, i.e., the readout of samples from a window thatthis is smaller than the entire array by appropriate control of the rowand column addresses.

In some embodiments, the light sensing devices 630 are incorporated intoan integrated circuit (“the sensor IC”). Furthermore, the samplingsubsystem 640 may include a readout integrated circuit (ROIC) that isconfigured for coupling to the sensor IC. In some embodiments, the ROICmay be a prior art ROIC provided by any of various manufacturers. TheROIC may include circuitry designed to read the signals generated by thelight sensing devices in the sensor IC. In one embodiment, the ROICincludes the circuitry shown in FIG. 20, minus the array of lightsensing devices. In another embodiment, the ROIC includes the circuitryshown in FIG. 20, minus the array of light sensing devices and minus theADCs. In another embodiment, the ROIC may include the circuitry of FIG.21, minus the array of light sensing devices. In another embodiment, theROIC may include the circuitry of FIG. 21, minus the array of lightsensing devices and minus the ADC.

The ROIC may be coupled to the sensor IC by “solder bump bonding”, awell known technique in the field of semiconductor assembly.

The ROIC may be implemented using CMOS technology. However, othertechnologies are possible.

In some embodiments, a sensor IC containing the array of light sensingdevices may be physically coupled to a transmissive realization of thelight modulation unit 110. In particular, the two may be coupled so thatthe backside (i.e., the light output side) of the transmissive lightmodulation unit is directly facing the light-sensing side of the sensorIC. Because the light sensing array is in close proximity to the arrayof light modulating elements, a high percentage of the light in eachspatial subset of the modulated light stream is delivered to itsrespective light sensing device.

In some embodiments, optical subsystem 620 may include one or morecomponents such as those illustrated in FIGS. 22A-F. FIG. 22A shows themodulated light stream MLS being separated into two sub-streams by amirror 2210M that blocks a portion (e.g., half) of the modulated lightstream. This principle generalizes to any number of mirrors. Forexample, FIG. 22B shows the modulated light stream being separated intothree sub-streams by mirrors 2210Ma and 2210Mb, each of which interceptsa corresponding portion of the modulated light stream. FIG. 22C shows anangled mirror 2210AM being used to separate the modulated light streaminto two sub-streams. FIG. 22D shows a prism 2210P being used toseparate the modulated light stream into two sub-stream by virtue oftotal internal reflection off of two surfaces of the prism. FIG. 22Eshows a prism 2210Q being used to separate the modulated light streaminto two sub-streams based on the refractive properties of the prism2210Q. FIG. 22F shows a reflective object 2210C having three reflectivefaces (or surfaces) that meet at a point P being used to separate themodulated light stream into three sub-streams. Representative rays R₁,R₂ and R₃ are shown.

Method 2300

In some embodiments, a method 2300 for operating on light may includethe actions shown in FIG. 23.

Action 2310 includes modulating an incident stream of light with asequence of spatial patterns in order to produce a modulated lightstream, e.g., as variously described above. The modulation is performedby an array of light modulating elements, e.g., as variously describedabove.

Action 2320 includes delivering light from each of a plurality ofspatial subsets of the modulated light stream onto a respective one of aplurality of light sensing devices, e.g., as variously described above.The spatial subsets of the modulated light stream are produced byrespective subsets of the array of light modulating elements. Eachsubset of the array of light modulating elements produces the respectivespatial subset of the modulated light stream by modulating a respectivespatial subset of the incident light stream. The operation of deliveringthe light from the spatial subsets to the light sensing device may beperformed by an optical subsystem, e.g., as variously described above.

Action 2330 includes generating electrical signals, where each of theelectrical signals is generated by a respective one of the light sensingdevices, e.g., as variously described above. Each of the electricalsignals represents intensity of a respective one of the spatial subsetsof the modulated light stream as a function of time.

Action 2340 includes acquiring samples of the electrical signals, e.g.,as variously described above. The samples include sample subsets thatcorrespond respectively to the electrical signals. Each sample subsetincludes a plurality of samples of the respective electrical signal.

In some embodiments, each of the sample subsets is usable to construct arespective sub-image of an image, where each sub-image represents arespective one of the spatial subsets of the incident light stream. Inother embodiments, the sample subsets may be too small in number ofsamples to construct sub-images, but may be used to perform any of avariety of inference tasks, as variously described above.

In some embodiments, the method 2300 also includes operating on thesample subsets to construct the respective sub-images.

In some embodiments, the method 2300 also includes transmitting thesample subsets to a receiver through a communication channel, e.g., asvariously described above.

As above, let m_(SS) denote the number of samples in each sample subset,N denote the number of the light modulating elements, L denote thenumber of the light sensing devices, n denote is the number of pixels inthe final image, where n is less than or equal to N, and let R_(P)denote the maximum rate (in spatial patterns per unit time) of patternmodulation by the light modulation unit 110. As described above, thenumber m_(n/L) of compressive sensing measurements required toreconstruct an n/L-pixel image decreases as L increases. Thus, the valueof L may be selected so that m_(SS)≧m_(n/L), and, m_(SS)/R_(P) is lessthan or equal to a target time of acquisition.

In some embodiments, the operation of acquiring the samples of theelectrical signals includes acquiring the samples in groups, e.g., asvariously described above. The groups may be acquired at a group rate ofR_(G) groups per unit time. Each of the groups includes at least onesample of each of the electrical signals. Each of the groups correspondsto a respective one of the spatial patterns. (In some embodiments, morethan one group is acquired per spatial pattern.) In some embodiments, Lis selected so that m_(SS)/R_(G) is less than or equal to a targetacquisition time.

Decreasing the Data Sampling Time in a Compressive-Imaging Device

One of the challenges in compressive imaging is acquiring the compressedimages fast enough to compete with prior art FPA-based cameras that canoperate at video rates. (FPA is an acronym for “focal plane array”. Afocal plane array is an image-sensing device that includes an array ofphotodetector elements, typically rectangular, and typically positionedat the focal plane of an optical system.) For at least certain values ofN, an N-element FPA can be used acquire N-pixel images at video rates.If N is large enough, it may be difficult for a compressive-sensingcamera, such as the camera of FIG. 1, to acquire compressedrepresentations of N-pixel images at video rates. The difficulty arisesbecause of the single photodiode. To reconstruct an N-pixel image withgiven accuracy requires a certain minimum number of samples of thephotodiode signal S(t) to be acquired. While that minimum number is muchsmaller than N, it is also significantly larger than one. For example,with a compression ratio of 1/10^(th), a compressive-imaging camerahaving a DMD with 1 million micromirrors may acquire a compressedrepresentation of an image of pixel count equal to 1 million bycapturing approximately 100,000 samples of the photodiode signal S(t),assuming each sample corresponds to a respective one of the spatialpatterns being applied by the DMD 40. If the DMD's maximum rate ofmodulation is 32,000 patterns per second, then the time required toacquire the compressed image representation is 100,000/32 kHz=3.125seconds, too slow to allow video acquisition.

The present patent discloses, among other things, a mechanism forreducing the acquisition time for a compressed image representation (orincreasing the frame rate for acquiring a sequence of compressed imagerepresentations) by introducing hardware parallelism into acompressive-imaging camera. See, e.g., FIG. 14. In one embodiment, thehardware parallelism involves replacing the single photodiode with fourlight sensing devices, and replacing the single ADC with four ADCs (oneADC per light sensing device). In addition, each light sensing devicereceives modulated light from a corresponding quadrant of the DMD, andthe samples acquired from each light sensing device are used toconstruct a corresponding sub-image of the final image. Because eachlight sensing device sees only ¼^(th) of the 1 million micromirrorarray, the sub-images are each of size (1 million)/4. Consequently, thenumber of samples per light sensing device required to reconstruct thesub-images decreases approximately by a factor of four, i.e., decreasesfrom 100,000 to 25,000. Assuming each ADC operates at the same samplerate as the ADC in the single photodiode system, the time for acquiringthe compressed image representation is reduced by a factor of four:25,000/32 kHz=0.78 seconds.

More generally, the single photodiode of FIG. 1 may be replaced with L>1light sensing devices, and the single ADC may be replaced with L lightsensing devices (one ADC per light sensing device). Each light sensingdevice receives modulated light from a corresponding region of anN-element light modulator (not necessarily a DMD and not necessarily N=1million). Assuming the L regions are equally sized, each light sensingdevice sees N/L elements of the light modulator. Thus, the L sub-imagesare each of pixel count N/L. As a first order approximation, the numberof samples per light sensing device required to reconstruct the Lsub-images decreases by a factor of L, assuming a linear relationshipbetween pixel count and number of sample required for reconstruction.Thus, the time required to acquire the compressed image representationis reduced by a factor of L. (Note, even if the relationship is notexactly linear, the number of samples required for the reconstruction ofeach sub-image nevertheless decreases significantly as L increases.Thus, the notion of achieving decreased acquisition time by increasing Lholds even if the relationship is not linear.) A system designer maythen select the value of L to achieve any desired acquisition time.

In one set of embodiments, a system 2400 for operating on light may beconfigured as shown in FIG. 24. The system 2400 may include the lightmodulation unit 110, the optical subsystem 620 and the plurality oflight sensing devices 630 as described above, and may also include aplurality of analog-to-digital converters (ADCs) 2410. (Furthermore, anysubset of the features, embodiments and elements discussed above andbelow may be incorporated into system 2400.)

The light modulation unit 110 is configured to modulate an incidentstream of light with a sequence of spatial patterns in order to producea modulated light stream. The light modulation unit includes an array oflight modulating elements as variously described above. In variousembodiments, the number of elements N in the array may vary over a widerange, e.g., as described above.

The optical subsystem 620 is configured to direct spatial subsets of themodulated light stream to respective ones of the light sensing devices,i.e., onto the light sensing surfaces of the light sensing devices. Thespatial subsets of the modulated light stream are produced by respectivenon-overlapping regions of the array of light modulating elements. Eachregion of the array of light modulating elements produces the respectivespatial subset of the modulated light stream by modulating a respectivespatial subset of the incident light stream. Each of the light sensingdevices is configured to generate a respective electrical signal thatrepresents intensity of the respective spatial subset of the modulatedlight stream as a function of time.

As discussed above, the expression “direct spatial subsets of themodulated light stream to respective light sensing devices” does notrequire that all the light from each spatial subset arrives at thecorresponding light sensing device, or that the light arriving at eachlight sensing device comes only from the respective spatial subset. Somelight loss and loss cross over is tolerated, e.g., depending on thedesired accuracy of image reconstruction, or the target manufacturingcost of the system 2400.

In some embodiments, the plurality of light sensing devices isincorporated into a single integrated circuit. In other embodiments, theplurality of light sensing devices may be discrete devices orseparately-packaged devices. In some embodiments, the light sensingdevices are arranged in an array. In other embodiments, the lightsensing devices are arranged in an irregular or non-uniform fashion, or,distributed to separate locations.

Each of the ADCs 2410 is configured to capture a respective sequence ofsamples of the electrical signal generated by a respective one of thelight sensing devices. The sequence of samples captured by each of theADCs is usable to construct a respective sub-image that represents thecorresponding spatial subset of the incident light stream.

In some embodiments, the optical subsystem may include a plurality oflenses, e.g., as variously described above. Each of the lenses isconfigured to direct a respective one of the spatial subsets of themodulated light stream onto a light sensing surface of a respective oneof the light sensing devices.

In some embodiments, the optical subsystem may include a fiber couplingdevice and a plurality of fiber bundles (or fibers), e.g., as variouslydescribed above. The fiber coupling device is configured to couple lightfrom each of the spatial portions of the modulated light stream onto arespective one of the fiber bundles. Each of the fiber bundles isconfigured to deliver the light from the respective spatial subset ofthe modulated light stream onto a light sensing surface of a respectiveone of the light sensing devices.

In some embodiments, the optical subsystem may include a focusing lens,e.g., as variously described above. The plurality of light sensingdevices may be located at an image plane of the focusing lens.

In some embodiments, the plurality of ADCs are configured to capturetheir respective sample sequences synchronously, e.g., in response to acommon sampling clock.

In some embodiments, each of the sample sequences has a sample sizem_(SS) that is smaller than the number of pixels n_(PSI) in therespective sub-image. The sample size of a sample sequence is the numberof samples included in the sample sequence. For example, in differentembodiments, the sample size m_(SS) may be, respectively, less than 70%of n_(PSI), less than 60% of n_(PSI), less than 50% of n_(PSI), lessthan 40% of n_(PSI), less than 30% of n_(PSI), less than 20% of n_(PSI),less than 10% of n_(PSI), less 5% of n_(PSI).

In some embodiments, the ADCs 2410 are configured to acquire the samplesequences over a time interval in which the incident light stream ismodulated with m_(P) of the spatial patterns. Furthermore, the ADCs 2410may acquire the sample sequences so that each sample sequence includesat least one sample for each of the mp spatial patterns. The number mpis less than the number n_(PSI) of pixels per sub-image. For example, indifferent embodiments, the number m_(P) is, respectively, less than 70%of n_(PSI), less than 60% of n_(PSI), less than 50% of n_(PSI), lessthan 40% of n_(PSI), less than 30% of n_(PSI)less than 20% of n_(PSI),less than 10% of n_(PSI), less than 5% of n_(PSI).

In one set of embodiments, a system 2500 for operating on light may beconfigured as shown in FIG. 25. System 2500 may include the lightmodulation unit 110 as variously described above, and may also includean optical subsystem 2520, an array of light sensing elements 2530 andcircuitry 2540. (Any subset of the various embodiments, features andelements described above may be incorporated into system 2500.)

The light modulation unit 110 is configured to modulate an incidentstream of light with a sequence of spatial patterns in order to producea modulated light stream. The light modulation unit includes an array ofN light modulating elements. The number N may take any of variousvalues, e.g., as variously described above and below. The number L oflight sensing elements in the light sensing array 2530 is less than Nand greater than one. The number L may take any of the various values,e.g., as variously described above and below.

In different sets of embodiments, the ratio N/L may be, respectively, inthe range [4,9], in the range [9,16], in the range [16,25], in the range[25,36], in the range [36,49], in the range [49,64], in the range[64,256], in the range [256,1024], in the range from 1024 to infinity.

The optical subsystem is configured to focus light from each of aplurality of spatial portions of the modulated light stream onto arespective one of the array of light sensing elements. The spatialportions of the modulated light stream are produced by respectivenon-overlapping regions of the array of light modulating elements. Eachregion of the array of light modulating elements produces the respectivespatial portion of the modulated light stream by modulating a respectivespatial portion of the incident light stream. The L light sensingelements are configured to generate respective electrical signals. Eachof the electrical signals represents intensity of the respective spatialportion of the modulated light stream as a function of time.

The circuitry 2540 is configured to read groups of samples from thearray of L light sensing elements. Each group of samples includes Lsamples, i.e., one sample from each of the L electrical signals. Inother words, each group includes one sample from each of the L lightsensing elements.

In some embodiments, the circuitry 2540 is configured to read the groupsat a maximum rate of R_(G) groups per unit time. Each of the groupscorresponds to a respective one of the spatial patterns. The sequence ofspatial patterns is configured so that M of said groups are usable toconstruct an N-pixel image representing the incident light stream.

In some embodiments, M is greater than one and less than or equal toN/L.

In some embodiments, M is less than N/L.

In some embodiments, the value of L is selected to achieve anacquisition time for the M groups that is less than or equal to a targetacquisition time.

In some embodiments, the circuitry 2540 is configured to read the groupsat a maximum rate of R_(G) groups per unit time, where each of thegroups corresponds to a respective one of the spatial patterns. Thesequence of spatial patterns is configured so that subsets of thegroups, each being M groups in length, are usable to constructrespective N-pixel images. M is greater than one and less than or equalto N/L. The integer L may be selected so that a rate of acquisition ofthe group subsets is greater than or equal to a target acquisition rate.

In some embodiments, the circuitry 2540 is included in a readoutintegrated circuit (ROIC), e.g., a ROIC as variously described above. Inone embodiment, the array of L light sensing elements is included in asecond integrated circuit. The readout integrated circuit may be bumpbonded to the second integrated circuit.

In some embodiments, the circuitry 2540 and the array of light sensingelements 2540 are included in a single integrated circuit. In oneembodiment, the single integrated circuit is implemented using CMOStechnology. For example, the single integrated circuit may be a CMOSimage sensor.

FPA-Based Compressive-Imaging Device

In one set of embodiments, the plurality of light sensing devices 640 ofsystem 600 are incorporated (integrated) in a focal plane array (FPA).At least in this context the light sensing devices may be referred to as“elements”. The FPA might have any of a wide variety of resolutions,e.g., any resolution of the form L_(x)×L_(y) with L_(x) and L_(y) beingintegers greater than or equal to two. Typical examples of FPAresolution might include 8×8, 16×16, 32×32, 64×64, 128×128, 256×256,320×256, 640×480, etc. In some embodiments, the rate R_(G) at whichsample groups can be read from the FPA may be slower (e.g., considerablyslower) than the maximum pattern modulation rate R_(P) of the lightmodulation unit 110, especially when the FPA is large. A sample groupmay include one sample for each of the light sensing elements in theFPA. Thus, for an FPA with resolution L_(x)×L_(y), each sample group mayinclude L=L_(x)*L_(y) samples.

FIG. 26A shows a conceptual diagram for one embodiment of an FPA-basedcompressive imaging system 2600. (Imaging system 2600 may also includeany subset of the features, embodiments and elements discussed above andbelow.)

The incident light stream is received by the optical subsystem 105(e.g., a camera lens) and focused upon the light modulation unit 110(e.g., a Texas Instruments DLP® micromirror device). The lightmodulation unit includes an array of N light modulating elements. Eachof the N light modulating elements may correspond to an image pixel inthe final reconstructed image, so that the pixel count of the finalreconstructed image may be determined by the element-count N of thelight modulation unit and not the element-count L of the FPA. The lightmodulation unit 110 receives the incident light stream from the opticalsubsystem 105 and modulates the incident light stream with sequence ofspatial patterns to obtain a modulated light stream MLS, e.g., asvariously described above. (In the case of a DLP, the incident lightstream is modulated by micromirrors.) The modulated light stream MLS maybe focused onto the FPA by the

optical subsystem 620, e.g., as variously described above. The FPA hasan element count L that is smaller than the element count N of the lightmodulation unit 110.

The readout integrated circuit (ROIC) reads sample groups from the FPA.The sample groups may be supplied to the computer system 2630 via ROICinterface unit 2620. In some embodiments, the FPA and ROIC may becombined to form a hybrid structure 2610, e.g., by coupling or bondingthe two integrated circuits together. The computer system 2630 mayoperate on a set of the sample groups to reconstruct an N-pixel pixelimage. In particular, the computer system 2630 may partition the samplesin the sample groups into L sample subsets corresponding respectively tothe L light sensing devices, and then reconstruct L sub-images of sizeN/L based on the respective sample subsets. The L sub-images are thenjoined together to form the final N-pixel image.

In some embodiments, the system 2600 may acquire sample groupscontinuously to support video acquisition.

The computer system 2630 may also supply control information to acontroller 2640 (e.g., an FPGA-based controller). The controller 2640controls the light modulation unit, e.g., supplies the sequence ofspatial patterns to the light modulation unit 110.

The optical subsystem 620 focuses the modulated light stream onto theFPA so that light from subsets of the array of light modulating elementsare received at respective light modulating elements of the FPA. Thesubsets may be non-overlapping and cover the array of light modulatingelements. Each subset may be a contiguous collection of the lightmodulating elements. For example, in one embodiment, the number of lightsensing elements L equals N/9, and the array of light modulatingelements is partitioned into 3×3 subsets. Each light sensing elementreceives light from a respective one of the 3×3 subsets. The samplesfrom each light sensing element are used to reconstruct a respective9-pixel sub-image. The N/9 sub-images of pixel count 9 are joinedtogether to form the final N-pixel image. Thus, in this embodiment, thesystem 2600 acquires images that have nine times the resolution of theFPA. Because the pixel count of each sub-image is so small (in thiscase, equal to 9), the number of samples of each light sensing devicerequired to reconstruct the corresponding 9-pixel sub-image is nine (orperhaps, more than nine in the presence of noise); one experiences nocompressive sensing gain. Thus, nine sample groups may be used toreconstruct the entire N-pixel image, and the frame rate (compressedimages per second) of the imaging system 2600 is nine times slower thanthe group rate R_(G) of the FPA.

The sequence of spatial patterns used to drive the light modulation unit110 may include nine (or more) spatial patterns. The nine spatialpatterns are designed so the restriction of the nine spatial patterns toeach subset of the light modulating array gives nine 3×3 patterns thatare usable to reconstruct the corresponding 9-pixel sub-image from thenine corresponding samples of the respective electrical signal. Forexample, in one embodiment, the nine 3×3 patterns correspond to the nineEuclidean basis elements on R⁹ (nine-dimensional Euclidean space.) Orequivalently, the nine 3×3 patterns each have a different one of thenine light sensing elements turned ON and the remaining eight elementsturned OFF. More generally, the nine 3×3 patterns may be defined by any9×9 invertible matrix, where each of the nine 3×3 patterns correspondsto a different row of the 9×9 invertible matrix.

In some embodiments, each of the nine spatial patterns is spatiallyperiodic, based on the periodic repetition of a corresponding 3×3pattern. Thus, nine 3×3 patterns may be used to generate the ninespatial patterns.

More generally, the array of light modulating elements may bepartitioned into subsets of size Q_(x)×Q_(y), where Q_(x) and Q_(y) arepositive integers such that Q=Q_(x)*Q_(y) is greater than or equal totwo. Thus, the number of light sensing elements in the FPA is L=N/Q.Each light sensing element receives light from a respective one of theQ_(x)×Q_(y) subsets. The samples from each light sensing element areused to reconstruct a respective Q-pixel sub-image. The N/Q sub-imagesof pixel count Q are joined together to form the final N-pixel image.Thus, the system 2600 acquires images that have Q times the resolutionof the FPA. If Q is small, the number of samples m of each light sensingdevice used to reconstruct the corresponding Q-pixel sub-image may be Qor close to Q (or more than Q in the presence of noise). As Q increases,the ratio m/Q decreases because one begins to see the benefit ofcompressive sensing gain. Thus, Q or fewer sample groups may be used toreconstruct the entire N-pixel image, and the ratio R_(G)/Q may be takenas a lower bound for the image frame rate (compressed images per second)of the imaging system 2600, where R_(G) is the group rate of the FPA. Ifoversampling by factor c_(OS>)1 is being employed the lower bound wouldbe inversely scaled: lower bound=R_(G)/(c_(OS)Q).

In some embodiments, the sequence of spatial patterns used to drive thelight modulation unit 110 may include m spatial patterns. The m spatialpatterns are designed so the restriction of the m spatial patterns toeach subset of the light sensing array gives m patterns of sizeQ_(x)×Q_(y) that are usable to reconstruct the corresponding Q-pixelsub-image from the m corresponding samples of the respective electricalsignal. If Q is small, e.g., less than 64, m may be set equal to Q, andthe m patterns of size Q_(x)×Q_(y) may correspond to the Q Euclideanbasis elements on R^(Q), or more generally, to the rows of any Q×Qinvertible matrix. If Q is not small, the m patterns of size Q_(x)×Q_(y)may correspond to the rows of any set of m vectors in R^(Q) that areincoherent relative to the reconstruction vector set to be used forreconstructing the Q_(x)×Q_(y) sub-image.

The table given in FIG. 26B shows possible image frame rates and imageresolutions for various FPA sizes, assuming that the FPA has a grouprate of 120 groups per second or 200 groups per second. (Frame ratescales with FPA group rate.) Additionally, as the number Q of lightmodulating elements per FPA element increases to 64 or more, framesrates can be increased by a factor of approximately two because thenumber of sample groups per image frame can be decreased. While thetable assumes the light modulation unit is a DMD, the results given aregenerally applicable to any type of light modulation unit having N lightmodulating elements.

In some embodiments, the imaging system 2600 can also be dynamicallyshifted between different settings of image-resolution vs. frame-rate.For example, using a 32×32 FPA, the imaging system can operate at either128×128 resolution and 7.5 FPS, or at 192×192 resolution and 3.3 FPS.(FPS is an acronym for “frame per second”.)

In some embodiments, the imaging system 2600 may perform imageprocessing on the sample groups prior to image reconstruction. Forexample, hot spots may be identified in the incident light field basedon analysis of the sample groups, and then the identified hot spots maybe attenuated or nulled by appropriate control of the spatial patterns.As another example, an object may be tracked based on an analysis of thesample groups. The result of the tracking may be used to controlactuators that adjust of the orientation of the camera, e.g., in realtime.

Method for Designing a CI Device when Pattern Rate Limited

In one set of embodiments, a computer-implemented method 2700 fordesigning a compressive-imaging device may involve the actions shown inFIG. 27A. The method may be performed by a computer system, e.g., acomputer system including one or more microprocessors.

At 2710, the computer system may receive information specifying a targeteffective rate R_(I) of acquisition of N-pixel images for thecompressive-imaging device, where N is a number of light modulatingelements in a light modulation unit to be included in thecompressive-imaging device. For example, the target effective rate maybe specified by user input.

The target rate is referred to as being an “effective” image acquisitionrate because the compressive-imaging device does not acquire the n-pixelimages directly. Rather the compressive-imaging device is configured toacquire sample sets that are usable to reconstruct the respectiven-pixel images. Until the n-pixel images are reconstructed from thesample sets, the n-pixel images exist only as transient patterns oflight that traverse the optical pathway of the compressive-imagingdevice.

At 2720, the computer system may determine (e.g., compute) a value foreach of variables R_(P) and L such that R_(P)/f(N/L) is greater than orequal to the target effective rate R_(I), where R_(P) is a maximumpattern modulation rate of the light modulation unit, where L is anumber of light sensing devices to be included in thecompressive-imaging device. The function f is a function that maps apositive integer value v to a corresponding number f(v), where f(v) isgreater than or equal to a minimum number of compressive sensing samplesrequired to reconstruct a v-pixel image.

The function f may have any of a wide variety of functional forms. Insome embodiments, the function f may be empirically derived. In someembodiments, the function f may depend on one or more factors such as:the targeted accuracy of reconstruction of the N-pixel images; the classof spatial patterns to be used to drive the light modulation unit; thesparsity-index value of the N-pixel images; etc.

In one embodiment, the function f may conform to the expressionf(N/L)=M _(N) /L,where M_(N) is greater than or equal to a minimum number of compressivesensing samples required to reconstruct a generic N-pixel image, e.g.,with given accuracy.

In another embodiment, the function f may have the form:f(v)=c*K _(N)*log(v),where c is a positive constant, where K_(N) is a sparsity index valueassociated with the generic N-pixel image.

In yet another embodiment, the function f may have the form:f(v)=c*K _(v)*log(v),where c is a positive constant, where K_(v) is a sparsity index valueassociated with a generic v-pixel image.

At 2730, the computer system may store the determined values of thevariables R_(P) and L in a memory. The determined values of R_(P) and Lat least partially specify a design for the compressive-imaging device.

In some embodiments, the method 2700 may also include displaying thedetermined value of the variable L and/or the determined value of thevariable R_(P).

In some embodiments, the action of determining a value for each of thevariables R_(P) and L includes: receiving input (e.g., user input)specifying the value for R_(P); and selecting a minimum value of thevariable L subject to the condition that R_(P)/f(N/L) is greater than orequal to rate R_(I).

In some embodiments, the action of determining a value for each of thevariables R_(P) and L includes: receiving input (e.g., user input)specifying the value for L, where the specified value of L is greaterthan one; and selecting a minimum value of the variable R_(P) subject tothe condition that R_(P)/f(n/L) is greater than or equal to rate R_(I).

In some embodiments, the function f depends on a selection of a class ofspatial patterns to be used by the compressive-imaging device. In oneembodiment, the method 2700 also includes receiving user inputspecifying the class of spatial patterns.

In some embodiments, the function f depends on a target accuracy withwhich the N-pixel images are to be reconstructed. In one embodiment, themethod 2700 also includes receiving user input specifying the targetaccuracy.

In some embodiments, the function f depends on a specified sparsityindex value K_(N) for the N-pixel images. In one embodiment, the method2700 includes receiving user input specifying the sparsity index valueK_(N).

In some embodiments, the method 2700 may also include constructing thecompressive-imaging device so that the compressive-imaging deviceincludes: the light modulation unit with the N light modulatingelements, and the determined number L of light sensing devices, whereeach of the determined number L of light sensing devices is configuredto receive light from a corresponding subset (or subregion) of the Nlight modulating elements, e.g., as variously described above.

Method 2750

In one set of embodiments, a computer-implemented method 2750 fordesigning a compressive-imaging device may involve the actions shown inFIG. 27B. The method may be performed by a computer system.

At 2760, the computer system may receive information (e.g., user input)specifying a target effective rate R_(I) of acquisition of n-pixelimages for the compressive-imaging imaging device, where n is less thanor equal to a number N of light modulating elements in a lightmodulation unit to be included in the compressive-imaging device.

At 2770, the computer system may determine (e.g., compute) a value foreach of variables R_(P), L and n such that R_(P)/f(n/L) is greater thanor equal to the target effective rate R_(I), where R_(P) is a maximumpattern modulation rate of the light modulation unit, where L is anumber of light sensing devices to be included in thecompressive-imaging device, where f is a function that maps a positiveinteger value v to a corresponding number f(v), where f(v) is greaterthan or equal to a minimum number of compressive sensing samplesrequired to reconstruct a v-pixel image. The function f may have any ofthe features or any combination of the features described above.

At 2780, the computer system may store the determined values of thevariables R_(P), L and n in a memory. In some embodiments, the method2750 may also include displaying one or more of the determined values,e.g., at least the determined value of the variable L.

In some embodiments, the action of determining a value for each of thevariables R_(P), L and n includes: receiving input (e.g., user input)specifying the values for R_(P), L and N; and selecting (or computing) amaximum value of the variable n subject to the condition thatR_(P)/f(n/L) is greater than or equal to rate R_(I) and the conditionthat n is less than or equal to N.

In some embodiments, the action of determining a value for each of thevariables R_(P), L and n includes: receiving input (e.g., user input)specifying the values for R_(P) and n; and selecting a minimum value ofthe variable L subject to the condition that R_(P)/f(n/L) is greaterthan or equal to rate R_(I).

In some embodiments, the action of determining a value for each of thevariables R_(P), L and n includes: receiving input specifying the valuesfor L and n, where the specified value of L is greater than one; andselecting a minimum value of the variable R_(P) subject to the conditionthat R_(P)/f(n/L) is greater than or equal to rate R_(I).

In some embodiments, the function f depends on a selection of a class ofspatial patterns to be used by the compressive-imaging device. In oneembodiment, the method 2750 includes receiving user input specifying theclass of spatial patterns.

In some embodiments, the function f depends on a target accuracy withwhich the n-pixel images are to be reconstructed. In one embodiment, themethod 2750 also includes receiving user input specifying the targetaccuracy.

In some embodiments, the function f depends on a specified sparsityindex value K_(n) for the n-pixel images. In one embodiment, the method2750 includes receiving user input specifying the sparsity index valueK_(n).

In some embodiments, the method 2750 may also include constructing thecompressive-imaging device so that the compressive-imaging deviceincludes: the light modulation unit with N light modulating elements,and the determined number L of light sensing devices. Each of thedetermined number L of light sensing devices is configured to receivelight from a corresponding subset (or subregion) of the N lightmodulating elements, e.g., as variously described above.

Method for Designing a CI Device when Limited by Sample Group Rate froman Array of Light Sensing Devices

In one set of embodiments, a computer-implemented method 2800 fordesigning a compressive-imaging device may involve the actions shown inFIG. 28A. The method 2800 may be performed by a computer system, e.g., acomputer system including one or more microprocessors.

At 2810, the computer system may receive information (e.g., user input)specifying a target effective rate R_(I) of acquisition of N-pixelimages for the compressive-imaging device, where N is a number of lightmodulating elements in a light modulation unit to be included in thecompressive-imaging device.

At 2820, the computer system may determine a value for each of variablesR_(G) and L such that R_(G)/f(N/L) is greater than or equal to R_(I),where R_(G) is a maximum number of sample groups per second to beacquired from an array of L light sensing devices to be included in thecompressive-imaging device, where each sample group by definitionincludes one sample from each of the L light sensing devices. Thefunction f is a function that maps a positive integer value v to acorresponding number f(v), where f(v) is greater than or equal to aminimum number of compressive sensing samples required to reconstruct av-pixel image. The function f may have any of the features or anycombination of the features described above. In one embodiment, thefunction f may be the identity function: f(v)=v. The identity functionmay be used, e.g., when n/L is small.

At 2830, the computer system may store the determined values of thevariables R_(G) and L in a memory. The determined values of R_(G) and Lat least partially specify a design for the compressive-imaging device.

In some embodiments, the method 2850 may also include displaying thedetermined value of the variable L and/or the determined value of thevariable R_(G).

In some embodiments, the action of determining a value for each of thevariables R_(G) and L may include: receiving input (e.g., user input)specifying the value for R_(G); and selecting a minimum value of thevariable L subject to the condition that R_(G)/f(N/L) is greater than orequal to rate R_(I).

In some embodiments, the action of determining a value for each of thevariables R_(G) and L may include: receiving input (e.g., user input)specifying the value for L, where the specified value of L is greaterthan one; and selecting a minimum value of the variable R_(G) subject tothe condition that R_(G)/f(N/L) is greater than or equal to rate R_(I).

In some embodiments, the function f depends on a selection of a class ofspatial patterns to be used by the compressive-imaging device. In oneembodiment, the method 2800 also includes receiving user inputspecifying the class of spatial patterns.

In some embodiments, the function f depends on a target accuracy withwhich the N-pixel images are to be reconstructed. In one embodiment, themethod 2800 also includes receiving user input specifying the targetaccuracy.

In some embodiments, the method 2800 may also include constructing(assembling or manufacturing) the compressive-imaging device so that thecompressive-imaging device includes: the light modulation unit with theN light modulating elements, and the array of L light sensing devices,where the value of L agrees with the determined value of L. Each of theL light sensing devices is configured to receive light from acorresponding subset (or subregion) of the N light modulating elements,e.g., as variously described above.

Method 2850

In one set of embodiments, a computer-implemented method 2850 fordesigning a compressive-imaging device may involve the actions shown inFIG. 28B. The method 2850 may be performed by a computer system.

At 2860, the computer system may receive information (e.g., user input)specifying a target effective rate R_(I) of acquisition of n-pixelimages for the compressive-imaging device, where n is less than or equalto N, where N is a number of light modulating elements in a lightmodulation unit to be included in the compressive-imaging device.

At 2870, the computer system may determine a value for each of variablesR_(G), L and n such that R_(G)/f(n/L) is greater than or equal to R_(I),where R_(G) is a maximum number of sample groups per second to beacquired from an array of L light sensing devices to be included in thecompressive-imaging device, where each sample group by definitionincludes one sample from each of the L light sensing devices. Thefunction f is a function that maps a positive integer value v to acorresponding number f(v), where f(v) is greater than or equal to aminimum number of compressive sensing samples required to reconstruct av-pixel image. The function f may have any of the features or anycombination of the features described above.

At 2880, the computer system may store the determined values of thevariables R_(G), L and n in a memory.

In some embodiments, the method 2850 may also include displaying one ormore of the determined values.

In some embodiments, the action of determining a value for each of thevariables R_(G), L and n may include: receiving input (e.g., user input)specifying the values for R_(G), L and N; and selecting a maximum valueof the variable n subject to the condition that R_(G)/f(n/L) is greaterthan or equal to rate R_(I) and the condition that n is less than orequal to N.

In some embodiments, the action of determining a value for each of thevariables R_(G), L and n may include: receiving input (e.g., user input)specifying the values for R_(G) and n; and selecting a minimum value ofthe variable L subject to the condition that R_(G)/f(n/L) is greaterthan or equal to rate R_(I).

In some embodiments, the action of determining a value for each of thevariables R_(G), L and n may include: receiving input (e.g., user input)specifying the values for L and n, where the specified value of L isgreater than one; and selecting a minimum value of the variable R_(G)subject to the condition that R_(G)/f(n/L) is greater than or equal torate R_(I).

In some embodiments, the function f depends on a selection of a class ofspatial patterns to be used by the compressive-imaging device. In oneembodiment, the method 2850 includes receiving user input specifying theclass of spatial patterns.

In some embodiments, the function f depends on a target accuracy withwhich the n-pixel images are to be reconstructed. In one embodiment, themethod 2850 also includes receiving user input specifying the targetaccuracy.

In some embodiments, the method 2850 may also include constructing thecompressive-imaging device so that the compressive-imaging deviceincludes: the light modulation unit with the N light modulatingelements, and the array of L light sensing devices, where the value of Lagrees with the determined value of L, where each of the L light sensingdevices is configured to receive light from a corresponding subset (orsubregion) of the N light modulating elements, e.g., as variouslydescribed above.

Parallel Light Modulation Units Operating on Subregions of the IncidentBeam

In one set of embodiments, a system 2900 for operating on light may beconfigured as shown in FIG. 29A. The system may include of apre-modulation optical subsystem 2910, a plurality of light modulationunits 110-1 through 110-G, a plurality of light sensing devices 130-2through 130G, and a plurality of analog-to-digital converters (ADCs)140-1 through 140-G. (In addition, system 2900 may include any subset ofthe features, embodiments and elements discussed above.)

The pre-modulation optical subsystem 2910 may be configured to receivean incident light stream and separate the incident light stream into aplurality of light substreams B₁, B₂, . . . , B_(G). The lightsubstreams correspond respectively to non-overlapping regions of theincident light stream. The regions may cover the cross section of theincident light stream. Each of the regions of the incident light streammay be a contiguous region (or a simply connected region as describedabove).

In some embodiments, the pre-modulation optical subsystem 2910 includesone or more mirrors. In some embodiments, the pre-modulation opticalsubsystem includes one or more prisms. In some embodiments, thepre-modulation optical subsystem includes one or more beam splitters. Insome embodiments, the pre-modulation optical subsystem includes anoptical device having a plurality of facets. In some embodiments, thepre-modulation optical subsystem includes one or more mirrors, and/or,one or more prisms, and/or, one or more beam splitters, and/or, one ormore faceted optical devices.

The light modulation units 110-1 through 110-G may be configured torespectively modulate the light substreams in order to generaterespective modulated light substreams MLS₁, MLS₂, . . . , MLS_(G). Eachof the light modulation units is configured to modulate the respectivelight substream with a respective sequence of spatial patterns in orderto produce the respective modulated light substream. Each of the lightmodulation units includes a respective array of light modulatingelements, e.g., as variously described above. Each of the lightmodulation units may be a copy of (or similar to) the light modulationunit 110, as variously described above.

In some embodiments, the sequences of spatial patterns are identicalsequences, i.e., each light modulation unit uses the same sequence ofspatial patterns. Alternatively, each of the sequences may be different.The light sensing devices 130-1 through 130-G may be configured torespectively receive the modulated light substreams and generaterespective electrical signals I₁(t), I₂(t), . . ., I_(G)(t). Theelectrical signal generated by each light sensing device representsintensity of the respective modulated light substream as a function oftime.

Each of the ADCs 140-1 through 140-G may be configured to configured tocapture a sequence of samples of the electrical signal generated by arespective one of the light sensing devices. The sample sequencecaptured by each ADC is usable to construct a respective sub-imagerepresenting a respective one of the regions of the incident lightstream. The sub-images may be joined together to form a final image thatrepresents the incident light stream over the time interval ofacquisition.

Since the final image is divided into sub-images, each sub-image can betreated independently, that is, measured and reconstructed in parallel.Since each sub-image has fewer pixels than the entire image, fewermeasurements are required, reducing data acquisition time.Reconstruction computations may be performed in parallel so thatcomputation time is not increased.

In some embodiments, the system 2900 may also include a plurality ofpost-modulation optical subsystems configured to direct light fromrespective ones of the modulated light substreams onto respective onesof the light sensing devices. For example, each optical subsystem may beconfigured as described above in connection with optical subsystem 117.

In some embodiments, each of the light sensing devices 130-1 through130-G is realized by a corresponding plurality (or array) of lightsensing elements. Thus, each post-modulation optical subsystem may beconfigured as described above in connection with optical subsystem 640in order to deliver light from subsets of the corresponding modulatedlight stream onto the respective light sensing elements, as variouslydescribed above.

FIG. 29B shows a special case of system 2900 in the case G=2.

FIG. 30A shows the premodulation optical subsystem 2910 being realizedby a mirror 2910M that is positioned so that it intercepts half of theincident light stream L. The other half passes unimpeded. Theintercepted half is reflected to light modulation unit 110B. The otherhalf is supplied to light modulation unit 110A. This principlegeneralizes to any number of mirrors. For example, FIG. 30B shows twomirrors 2910Ma and 2910Mb uses to separate the incident light stream Linto three portions. Also, a set of two or more mirrors may be used toachieve multiple stages of separation. For example, the sub-stream B₁ inFIG. 30B may be further separated by one or more additional mirrors.

FIG. 31A shows premodulation optical subsystem 2910 being realized by anangled mirror 2910AM configured to separate the incident light streaminto two light sub-streams. FIG. 31B shows an alternative embodimentwhere prism 2910P separates the incident light beam based on totalinternal reflection at two surfaces of the prism. FIG. 31 C showsanother alternative based on a prism 2910Q.

FIG. 32A shows premodulation optical subsystem 2910 being realized by anobject (or device) 2910C having reflective faces (or facets) that meetat a point P. For example, a cube having three mirrored faces meeting atpoint P may be used to separate the incident light stream L into threelight sub-streams. Representative rays R₁, R₂ and R₃ are shown beingreflected from corresponding faces. FIG. 32B shows the pattern ofseparation as seen along the axis of the incident light stream. Thisprinciple of separation generalizes to objects having more than threefaces meeting at a point. For example, a tetrahedron has four facesmeeting an apex, and thus, may be used to separate the incident lightstream into four sub-streams.

In one set of embodiments, a method 3300 for operating on light mayinvolve the actions shown in FIG. 33.

Action 3310 includes separating an incident light stream into aplurality of light substreams, e.g., as variously described above. Thelight substreams correspond respectively to non-overlapping regions ofthe incident light stream, e.g., as variously described above.

Action 3320 includes modulating the light substreams in order to producerespective modulated light substreams. Each of the light substreams ismodulated by a respective one of a plurality of light modulation units,e.g., light modulation units as variously described above. Each of thelight modulation units includes a corresponding array of lightmodulating elements. The action of modulating the light substreamsincludes modulating each of the light substreams with a respectivesequence of spatial patterns (e.g., as variously described above) inorder to produce the respective modulated light substream.

Action 3330 includes directing light from each of the modulated lightsubstreams onto a respective one of a plurality of light sensing devices(e.g., light sensing devices as variously described above). Varioustechniques for directing light are described above.

Action 3340 includes generating electrical signals, e.g., as variouslydescribed above. Each of the electrical signals is generated by arespective one of the light sensing devices. The electrical signalgenerated by each light sensing device represents intensity of therespective modulated light stream as a function of time.

Action 3350 includes acquiring from each of the light sensing devices arespective sequence of samples of the respective electrical signal. Eachsample sequence is usable to construct a respective sub-imagerepresenting a respective one of the regions of the incident lightstream.

Pre-Modulation Separation of the Incident Light Stream into MultipleStreams that Carry the Same Image as the Incident Light Stream

In one set of embodiments, a system 3400 may be configured as shown inFIG. 34A. System 3400 includes an optical separation unit 3410, lightmodulation units 110A and 100B, light sensing devices 130A and 130B, andADCs 140A and 140B. (In addition, system 3400 may include any subset ofthe features, embodiments and elements discussed above in connectionwith system 100, system realization 200 and system 600.)

Optical separation unit 3410 separates the incident light stream L intotwo light streams B₁ and B₂. Each of the two light streams B₁ and B₂ maycarry the same image carried by the incident light stream. In otherwords, each of the two light streams B1 and B₂ may carry a copy (atlower intensity) of the image present in the modulated light stream(perhaps up to a transformation such as mirror reflection or scaling).

In some embodiments, approximately 50% of the incident light power goesto each of streams B₁ and B₂. The light modulation units 110A and 110Brespectively modulate the streams B₁ and B₂ with respective sequences ofspatial patterns in order to obtain respective modulated light streamsMLS₁ and MLS₂. The light sensing devices 130A and 130B respectivelygenerate electrical signals I₁(t) and I₂(t). Each electrical signalrepresents intensity of the respective modulated light stream as afunction of time. The ADCs 140A and 140B acquire samples of therespective electrical signals. Each ADC acquires a respective sequenceof samples of the respective electrical signal. In other words, ADC 140Aacquires a sequence of samples {I₁(k)} of the electrical signal I₁(t),and ADC 140B acquires a sequence of samples {I₂(k)} of the electricalsignal I₂(t). The two sample sequences may be used to construct an imagerepresenting the incident light stream L over the time interval ofacquisition. (The image construction operation may be performed bysystem 3400 or by some other remote system.) If the image carried bylight stream B_(j), j=1 or 2, is related to the image in the incidentlight stream by a given transformation (e.g., mirror reflection), theimage construction algorithm may be configured to account for thattransformation, e.g., by applying the inverse transformation to thecorresponding sequence of spatial patterns, i.e., to the spatialpatterns that produce the sample sequence {I_(j)(k)}.

In some embodiments, each light modulation unit may have the same numberN of light modulating elements. (However, alternative embodiments arecontemplated where this is not the case.) Thus, the reconstructed imagemay be of pixel count N or less. Recall that a light modulating elementwith N light modulating elements may be operated at a lower effectiveresolution n<N, if desired.

The two post-separation branches (each comprising a light modulationunit, light sensing device and ADC) may operate in parallel. Forexample, each light modulation unit may modulate its respective lightstream with spatial patterns at a common pattern rate, e.g., beingdriven by a common pattern clock. Furthermore, each ADC may acquiresamples at a common sampling rate.

Because system 3400 has two post-separation branches, with each branchacquiring compressive-sensing samples (inner products) of the sameimage, the amount of time required to acquire the compressedrepresentation of the image is decreased by a factor of two, providedeach light modulation unit uses different spatial patterns. In otherwords, the two sequences of spatial patterns preferably have no (or onlya small percentage of) spatial patterns in common.

In alternative embodiments, light streams B₁ and B₂ carry differentspectral portions of the image carried by the incident light stream.

FIG. 34B shows the optical separation unit 3410 being realized as a beamsplitter 3410BS. However, a wide variety of other realizations arepossible. For example, in other embodiments, optical separation unit3410 may be realized by one or more of the following: branchedfiber-optic bundles, color separation prisms, holographic opticalelements, diffraction gratings, Dammann gratings and Fourier arraygenerators.

System 3500

In one set of embodiments, a system 3500 may be configured as shown inFIG. 35. System 3500 includes a pre-modulation optical separation unit3510, light modulation units 110-1 through 110-G, light sensing devices130-1 through 130-G, and ADCs 140-1 through 140-G. (In addition, system3500 may include any subset of the features, embodiments and elementsdiscussed above in connection with system 100, system realization 200,system 600 and system 3400.)

Optical separation unit 3510 separates the incident light stream L intolight streams B₁ through B_(G). However, the separation is not aseparation into spatial portions as is the case with pre-modulationoptical subsystem 2910. Instead, each light stream B_(j) carries (atlower power) the same image present in the incident light stream L.

In some embodiments, each light stream B_(j) receives approximately thesame fraction (1/G) of the power present in the incident light stream.

The light modulation units 110-1 through 110-G respectively modulate thestreams B₁ through B_(G) with respective sequences of spatial patternsin order to obtain respective modulated light streams MLS₁ throughMLS_(G). The light sensing devices 130-1 through 130-G respectivelygenerate electrical signals I₁(t) through I_(G)(t). Each electricalsignal I_(j)(t) represents intensity of the respective modulated lightstream MLS_(j) as a function of time. The ADCs 140-1 through 140-Gacquire samples of the respective electrical signals I₁(t) throughI_(G)(t). Each ADC acquires a respective sequence of samples of therespective electrical signal. In other words, ADC 140-j acquires asequence of samples {I_(j)(k)} of the electrical signal I_(j)(t). The Gsample sequences may be used to construct an image representing theincident light stream L over the time interval of acquisition. (Theimage construction operation may be performed by system 3500 or by someother remote system.) If the image carried by light stream B_(j) isrelated to the image in the incident light stream by a giventransformation (e.g., mirror reflection), the image constructionalgorithm may be configured to account for that transformation, e.g., byapplying the inverse transformation to the corresponding sequence ofspatial patterns, i.e., the spatial patterns that correspond the samplesequence {I_(j)(k)}. (The transformations may be known by calibration.)The image may be constructed using the union of the sample sequences{I₁(k)}, {I₂(k)}, {I_(G)(k)} and the union of the corresponding sets of(possibly transformed) spatial patterns.

The system 3500 preferably operates in a compressive fashion, where thetotal number of samples in the union of the samples sets {I₁(k)},{I₂(k)}, {I_(G)(k)} is smaller than the number of pixels N in theconstructed image.

In some embodiments, each light modulation unit may have the same numberN of light modulating elements. (However, alternative embodiments arecontemplated where this is not the case.) Thus, the reconstructed imagemay be of pixel count N or less. Recall that a light modulating elementwith N light modulating elements may be operated at a lower effectiveresolution n<N, if desired.

The G post-separation branches (each comprising a light modulation unit,light sensing device and ADC) may operate in parallel. For example, eachlight modulation unit may modulate its respective light stream withspatial patterns at a common pattern rate, e.g., being driven by acommon pattern clock. Furthermore, each ADC may acquire samples at acommon sampling rate, perhaps driven by a common sampling clock.

Because system 3500 has G post-separation branches, with each branchacquiring compressive-sensing samples (inner products) of the sameimage, the amount of time required to acquire the compressedrepresentation of the image is decreased by a factor of G, provided eachlight modulation unit uses different spatial patterns. In other words,the set of spatial patterns used by any one of the light modulationunits and the set of spatial patterns used by any other of the lightmodulation unit are preferably disjoint sets or sets having only a smallpercentage of overlap (intersection).

In some embodiments, system 3500 includes the optical subsystem 105 asdescribed above. The optical subsystem 105 may positioned before theoptical separation unit 3510. In other words, the incident light streamL passes through the optical subsystem 105 before encountering theoptical separation unit. The optical subsystem 105 and the lightmodulation units 110-1 through 110-G may be positioned so that therespective light streams B₁ through B_(G) are in focus at the respectivelight modulation units. Alternatively, or additionally, system 3500 mayinclude optical subsystems interposed between optical separation unit3510 and the respective light modulation units.

In some embodiments, any (or all) of the post-separation branches mayinclude the optical subsystem 117 (in any of its various embodiments) orthe optical subsystem 620 (in any of its various embodiments)intervening between the light modulation and the light sensing device ofthat branch.

Various Embodiments of Systems and Methods

Embodiments of various systems and methods are described in thefollowing numbered paragraphs.

1.1. A system comprising:

a light modulation unit configured to modulate an incident stream oflight with a sequence of spatial patterns in order to produce amodulated light stream, wherein the light modulation unit includes anarray of light modulating elements;

a plurality of light sensing devices;

an optical subsystem configured to deliver light from each of aplurality of spatial subsets of the modulated light stream onto arespective one of the plurality of light sensing devices, wherein thespatial subsets of the modulated light stream are produced by respectivesubsets of the array of light modulating elements, wherein each subsetof the array of light modulating elements produces the respectivespatial subset of the modulated light stream by modulating a respectivespatial subset of the incident light stream, wherein the light sensingdevices are configured to generate respective electrical signals,wherein each of the electrical signals represents intensity of therespective spatial subset of the modulated light stream as a function oftime; and

a sampling subsystem configured to acquire samples of the electricalsignals, wherein the samples include sample subsets that correspondrespectively to the electrical signals, wherein each sample subsetincludes a plurality of samples of the respective electrical signal.

1.2 The system of paragraph 1.1, wherein each of the sample subsets isusable to construct a respective sub-image of an image, wherein eachsub-image represents a respective one of the spatial subsets of theincident light stream.

1.3. The system of paragraph 1.2, wherein N is the number of the lightmodulating elements, wherein L is the number of the light sensingdevices, wherein n is the number of pixels in said image, wherein n isless than or equal to N, wherein the light modulation unit is configuredto modulate the incident light stream with the spatial patterns at amaximum rate of R_(P) spatial patterns per unit time, wherein m_(SS) isthe number of samples in each sample subset, wherein a minimum numberm_(n/L) of compressive-sensing samples required to reconstruct ann/L-pixel image with given accuracy decreases as L increases, whereinm_(SS) and L are selected so that m_(SS) is greater than or equal tom_(n/L), and m_(SS)/R_(P) is less than or equal to a target acquisitiontime.

1.4 The system of paragraph 1.2, wherein N is the number of the lightmodulating elements, wherein L is the number of the light sensingdevices, wherein n is the number of pixels in said image, wherein n isless than or equal to N, wherein each sample subset is usable toconstruct the respective sub-image with n/L pixels, wherein the samplingsubsystem is configured so that the number of samples in each samplesubset is dynamically programmable to allow different tradeoffs betweenimage quality and acquisition time of the samples of the electricalsignals.

1.5. The system of paragraph 1.2, further comprising a processing unitconfigured to operate on each sample subset to construct the respectivesub-image.

1.6 The system of paragraph 1.1, wherein the sampling subsystem isconfigured to acquire the samples of the electrical signals in groups,wherein each of the groups includes at least one sample of each of theelectrical signals, wherein each of the groups corresponds to arespective one of the spatial patterns.

1.7. The system of paragraph 1.6, wherein said sampling subsystem isconfigured to acquire the groups of samples of the electrical signals ata maximum group rate of R_(G) groups per unit time, wherein m_(SS) isthe number of samples in each sample subset, wherein N is the number ofthe light modulating elements, wherein n is less than or equal to N,wherein L is the number of the light sensing devices, wherein a numberm_(n/L) of the compressive sensing samples required to reconstruct ann/L-pixel image decreases as L increases, wherein L is selected so thatm_(SS) is greater than or equal to m_(n/L), and, m_(SS)/R_(G) is lessthan or equal to a target acquisition time.

1.8. The system of paragraph 1.1, wherein the sampling subsystemincludes a plurality of analog-to-digital converters (ADCs), whereineach of the ADCs is configured to acquire the samples of the electricalsignal generated by a respective one of the light sensing devices.

1.9. The system of paragraph 1.8, wherein at least a first and a secondof the ADCs have different bit resolutions.

1.10. The system of paragraph 1.1, wherein the optical subsystemincludes a plurality of lenses, wherein each of the lenses is configuredto direct the light from a respective one of the spatial subsets of themodulated light stream onto a light sensing surface of a respective oneof the light sensing devices.

1.11. The system of paragraph 1.1, wherein the optical subsystemincludes a fiber coupling device and a plurality of fiber bundles,wherein the fiber coupling device is configured to couple the light fromeach of the spatial subsets of the modulated light stream onto arespective one of the fiber bundles.

1.12. The system of paragraph 1.1, wherein the optical subsystemincludes a plurality of light pipes whose internal surfaces arereflective.

1.13. The system of paragraph 1.1, wherein the optical subsystemincludes a focusing lens, wherein the plurality of light sensing devicesare located at an image plane of the focusing lens.

1.14.The system of paragraph 1.1, wherein the subsets of the array ofthe light modulating elements are contiguous subsets.

1.15. The system of paragraph 1.1, wherein at least a first and a secondof the light sensing devices have different light sensitivities.

1.16. The system of paragraph 1.1, wherein the plurality of lightsensing devices are arranged in an array.

1.17 The system of paragraph 1.16, wherein the array is atwo-dimensional rectangular array, wherein the sampling subsystemincludes one analog-to-digital converter per column of the array.

1.18. The system of paragraph 1.1, wherein the plurality of lightsensing devices are incorporated into a first integrated circuit.

1.19. The system of paragraph 1.18, wherein the sampling subsystemincludes a readout integrated circuit, wherein the readout integratedcircuit is coupled to the first integrated circuit.

1.20. The system of paragraph 1.19, wherein the readout integratedcircuit is bump bonded to the first integrated circuit.

1.21. A method comprising:

modulating an incident stream of light with a sequence of spatialpatterns in order to produce a modulated light stream, where saidmodulating is performed by an array of light modulating elements;

delivering light from each of a plurality of spatial subsets of themodulated light stream onto a respective one of a plurality of lightsensing devices, wherein the spatial subsets of the modulated lightstream are produced by respective subsets of the array of lightmodulating elements, wherein each subset of the array of lightmodulating elements produces the respective spatial subset of themodulated light stream by modulating a respective spatial subset of theincident light stream;

generating electrical signals, wherein each of the electrical signals isgenerated by a respective one of the light sensing devices, wherein eachof the electrical signals represents intensity of a respective one ofthe spatial subsets of the modulated light stream as a function of time;and

acquiring samples of the electrical signals, wherein the samples includesample subsets that correspond respectively to the electrical signals,wherein each sample subset includes a plurality of samples of therespective electrical signal.

1.22 The method of paragraph 1.21, wherein each of the sample subsets isusable to construct a respective sub-image of an image, wherein eachsub-image represents a respective one of the spatial subsets of theincident light stream.

1.23. The method of paragraph 1.22, wherein N is the number of the lightmodulating elements, wherein L is the number of the light sensingdevices, wherein n is the number of pixels in said image, wherein n isless than or equal to N, wherein the incident light stream is modulatedwith the spatial patterns at a maximum rate of R_(P) spatial patternsper unit time, wherein m_(SS) is the number of samples in each samplesubset, wherein a minimum number m_(n/L) of compressive-sensing samplesrequired to reconstruct an n/L-pixel image with given accuracy decreasesas L increases, wherein m_(SS) and L are selected so that m_(SS) isgreater than or equal to m_(n/L), and m_(SS)/R_(P) is less than or equalto a target acquisition time.

1.24 The method of paragraph 1.21, further comprising: operating on thesample subsets to construct the respective sub-images.

1.25 The method of paragraph 1.21, further comprising: transmitting thesample subsets to a receiver through a communication channel.

1.26 The method of paragraph 1.21, wherein said acquiring the samples ofthe electrical signals includes acquiring the samples in groups, whereineach of the groups includes at least one sample of each of theelectrical signals, wherein each of the groups corresponds to arespective one of the spatial patterns.

1.27. The method of paragraph 1.26, wherein the groups are acquired at arate of R_(G) groups per unit time, wherein N is the number of the lightmodulating elements, wherein n is less than or equal to N, wherein L isthe number of the light sensing devices, wherein m_(SS) is the number ofsamples in each sample subset, wherein a number m_(n/L) of thecompressive sensing samples required to reconstruct an n/L-pixel imagedecreases as L increases, wherein L is selected so that m_(SS) isgreater than or equal to m_(n/L), and, m_(SS)/R_(G) is less than orequal to a target acquisition time.

2.1. A system comprising:

a light modulation unit configured to modulate an incident stream oflight with a sequence of spatial patterns in order to produce amodulated light stream, wherein the light modulation unit includes anarray of light modulating elements;

a plurality of light sensing devices;

an optical subsystem configured to direct spatial subsets of themodulated light stream to respective ones of the light sensing devices,wherein the spatial subsets of the modulated light stream are producedby respective non-overlapping regions of the array of light modulatingelements, wherein each region of the array of light modulating elementsproduces the respective spatial subset of the modulated light stream bymodulating a respective spatial subset of the incident light stream,wherein each of the light sensing devices is configured to generate arespective electrical signal that represents intensity of the respectivespatial subset of the modulated light stream as a function of time;

a plurality of analog-to-digital converters (ADCs), wherein each of theADCs is configured to capture a respective sequence of samples of theelectrical signal generated by a respective one of the light sensingdevices, wherein the sequence of samples captured by each of the ADCs isusable to construct a respective sub-image that represents thecorresponding spatial subset of the incident light stream.

2.2. The system of paragraph 2.1, wherein the optical subsystem includesa plurality of lenses, wherein each of the lenses is configured todirect a respective one of the spatial subsets of the modulated lightstream onto a light sensing surface of a respective one of the lightsensing devices.

2.3. The system of paragraph 2.1, wherein the optical subsystem includesa fiber coupling device and a plurality of fiber bundles, wherein thefiber coupling device is configured to couple light from each of thespatial portions of the modulated light stream onto a respective one ofthe fiber bundles, wherein each of the fiber bundles is configured todeliver the light from the respective spatial subset of the modulatedlight stream onto a light sensing surface of a respective one of thelight sensing devices.

2.4. The system of paragraph 2.1, wherein the plurality of light sensingdevices are incorporated into a single integrated circuit.

2.5. The system of paragraph 2.1, wherein the optical subsystem includesa focusing lens, wherein the plurality of light sensing devices arelocated at an image plane of the focusing lens.

2.6.The system of paragraph 2.1, wherein the plurality of ADCs areconfigured to capture their respective sample sequences synchronously.

2.7. The system of paragraph 2.1, wherein the plurality of light sensingdevices are arranged in an array.

2.8. The system of paragraph 2.1, wherein each of the sample sequenceshas a sample size that is smaller than the number of pixels in therespective sub-image.

3.1. A system comprising:

a light modulation unit configured to modulate an incident stream oflight with a sequence of spatial patterns in order to produce amodulated light stream, wherein the light modulation unit includes anarray of N light modulating elements;

an array of L light sensing elements, wherein L is greater than one butless than N;

an optical subsystem configured to focus light from each of a pluralityof spatial portions of the modulated light stream onto a respective oneof the array of light sensing elements, wherein the spatial portions ofthe modulated light stream are produced by respective non-overlappingregions of the array of light modulating elements, wherein each regionof the array of light modulating elements produces the respectivespatial portion of the modulated light stream by modulating a respectivespatial portion of the incident light stream, wherein the L lightsensing elements are configured to generate respective electricalsignals, wherein each of the electrical signals represents intensity ofthe respective spatial portion of the modulated light stream as afunction of time; and

circuitry configured to read groups of samples from the array of L lightsensing elements, wherein each group of samples includes L samples,wherein the L samples of each group include one sample for each of theelectrical signals.

3.2. The system of paragraph 3.1, wherein the circuitry is configured toread the groups at a maximum rate of R_(G) groups per unit time, whereineach of the groups corresponds to a respective one of the spatialpatterns, wherein the sequence of spatial patterns is configured so thatM of said groups are usable to construct an N-pixel image representingthe incident light stream, wherein M is greater than one and less thanor equal to N/L.

3.3. The system of paragraph 3.2, wherein L is selected to achieve anacquisition time for the M groups that is less than or equal to a targetacquisition time.

3.4. The system of paragraph 3.2, wherein M is less than N/L.

3.5. The system of paragraph 3.1, wherein the circuitry is configured toread the groups at a maximum rate of R_(G) groups per unit time, whereineach of the groups corresponds to a respective one of the spatialpatterns, wherein the sequence of spatial patterns is configured so thatsubsets of the groups, each subset of groups being M groups in length,are usable to construct respective N-pixel images, wherein M is greaterthan one and less than or equal to N/L, wherein L is selected so that arate of acquisition of the group subsets is greater than or equal to atarget acquisition rate.

3.6. The system of paragraph 3.1, wherein the circuitry is included in areadout integrated circuit, wherein the array of L light sensingelements are included in a second integrated circuit.

3.7. The system of paragraph 3.6, wherein the readout integrated circuitis bump bonded to the second integrated circuit.

3.8. The system of paragraph 3.1, wherein the circuitry and the lightsensing elements are included in a single integrated circuit.

4.1. A computer-implemented method for designing a compressive-imagingdevice, the method comprising:

utilizing a computer to perform: receiving information specifying atarget effective rate R_(I) of acquisition of N-pixel images for thecompressive-imaging device, wherein N is a number of light modulatingelements in a light modulation unit to be included in thecompressive-imaging device; determining a value for each of variablesR_(P) and L such that R_(P)/f(N/L) is greater than or equal to thetarget effective rate R_(I), wherein R_(P) is a maximum patternmodulation rate of the light modulation unit, wherein L is a number oflight sensing devices to be included in the compressive-imaging device,wherein f is a function that maps a positive integer value v to acorresponding number f(v), wherein f(v) is greater than or equal to aminimum number of compressive sensing samples required to reconstruct av-pixel image; and storing the determined values of the variables R_(P)and L in a memory.

4.2 The method of paragraph 4.1, further comprising displaying one ormore of the determined values.

4.3. The method of paragraph 4.1, wherein said determining a value foreach of the variables R_(P) and L includes: receiving input specifyingthe value for the

variable R_(P); and selecting a minimum value of the variable L subjectto the condition that R_(P)/f(N/L) is greater than or equal to rateR_(I).

4.4. The method of paragraph 4.1, wherein said determining a value foreach of the variables R_(P) and L includes: receiving input specifyingthe value for the variable L; and selecting a minimum value of thevariable R_(P) subject to the condition that R_(P)/f(N/L) is greaterthan or equal to rate R_(I).

4.5. The method of paragraph 4.1, wherein the function f depends on aselection of a class of spatial patterns to be used by thecompressive-imaging device.

4.6. The method of paragraph 4.1, wherein the function f depends on atarget accuracy with which the N-pixel images are to be reconstructed.

4.7. The method of paragraph 4.6 further comprising receiving user inputspecifying the target accuracy.

4.8. The method of paragraph 4.1, further comprising: constructing thecompressive-imaging device so that the compressive-imaging deviceincludes: the light modulation unit with the N light modulatingelements, and the determined number L of light sensing devices, whereineach of the determined number L of light sensing devices is configuredto receive light from a corresponding subset of the N light modulatingelements.

5.1. A method for designing a compressive-imaging device, the methodcomprising:

utilizing a computer to perform: receiving information specifying atarget effective rate R_(I) of acquisition of N-pixel images for thecompressive-imaging device, wherein N is a number of light modulatingelements in a light modulation unit to be included in thecompressive-imaging device; determining a value for each of variablesR_(G) and L such that R_(G)/f(N/L) is greater than or equal to R_(I),wherein R_(G) is a maximum number of sample groups per second to beacquired from an array of L light sensing devices to be included in thecompressive-imaging device, wherein each sample group by definitionincludes one sample from each of the L light sensing devices, wherein fis a function that maps a positive integer value v to a correspondingnumber f(v), wherein f(v) is greater than or equal to a minimum numberof compressive sensing samples required to reconstruct a v-pixel image;and storing the determined values of the variables R_(G) and L in amemory.

5.2 The method of paragraph 5.1, further comprising displaying one ormore of the determined values.

5.3. The method of paragraph 5.1, wherein said determining a value foreach of the variables R_(G) and L includes: receiving input specifyingthe value for the variable R_(G); and selecting a minimum value of thevariable L subject to the condition that R_(G)/f(N/L) is greater than orequal to rate R_(I).

5.4. The method of paragraph 5.1, wherein said determining a value foreach of the variables R_(G) and L includes: receiving input specifyingthe value for the variable L; and selecting a minimum value of thevariable R_(G) subject to the condition that R_(G)/f(N/L) is greaterthan or equal to rate R_(I).

5.5. The method of paragraph 5.1, wherein the function f depends on aselection of a class of spatial patterns to be used by thecompressive-imaging device.

5.6. The method of paragraph 5.1, wherein the function f depends on atarget accuracy with which the N-pixel images are to be reconstructed

5.7. The method of paragraph 5.6 further comprising receiving user inputspecifying the target accuracy.

5.8. The method of paragraph 5.1, further comprising constructing thecompressive-imaging device so that the compressive-imaging deviceincludes: the light modulation unit with the N light modulatingelements, and the array of L light sensing devices, wherein the value ofL agrees with the determined value of L, wherein each of the L lightsensing devices is configured to receive light from a correspondingsubset of the N light modulating elements.

6.1. A system comprising:

a pre-modulation optical subsystem configured to receive an incidentlight stream and separate the incident light stream into a plurality oflight substreams, wherein the light substreams correspond respectivelyto non-overlapping regions of the incident light stream;

a plurality of light modulation units configured to respectivelymodulate the light substreams in order to generate respective modulatedlight substreams, wherein each of the light modulation units isconfigured to modulate the respective light substream with a respectivesequence of spatial patterns in order to produce the respectivemodulated light substream, wherein each of the light modulation unitsincludes a respective array of light modulating elements;

a plurality of light sensing devices configured to respectively receivethe modulated light substreams and generate respective electricalsignals, wherein the electrical signal generated by each light sensingdevice represents intensity of the respective modulated light substreamas a function of time;

a plurality of analog-to-digital converters (ADCs), wherein each of theADCs is configured to configured to capture a sequence of samples of theelectrical signal generated by a respective one of the light sensingdevices, wherein the sample sequence captured by each ADC is usable toconstruct a respective sub-image representing a respective one of theregions of the incident light stream.

6.2 The system of paragraph 6.1, further comprising: a plurality ofpost-modulation optical subsystems configured to direct light fromrespective ones of the modulated light substreams onto respective onesof the light sensing devices.

6.3 The system of paragraph 6.1, wherein a number of the samples in eachsample sequence is smaller than a number of pixels in the respectivesub-image.

6.4 The system of paragraph 6.1, wherein the pre-modulation opticalsubsystem includes one or more mirrors.

6.6 The system of paragraph 6.1, wherein the pre-modulation opticalsubsystem includes a beam splitter.

6.7 The system of paragraph 6.1, wherein the pre-modulation opticalsubsystem includes an optical device having a plurality of facets.

6.8 The system of paragraph 6.1, wherein the sequences of spatialpatterns are identical sequences.

6.9 The system of paragraph 6.1, wherein each of the regions of theincident light stream is a contiguous region.

6.10. A method comprising:

separating an incident light stream into a plurality of lightsubstreams, wherein the light substreams correspond respectively tonon-overlapping regions of the incident light stream;

modulating the light substreams in order to produce respective modulatedlight substreams, wherein each of the light substreams is modulated by arespective one of a plurality of light modulation units, wherein saidmodulating the light substreams includes modulating each of the lightsubstreams with a respective sequence of spatial patterns in order toproduce the respective modulated light substream;

directing light from each of the modulated light substreams onto arespective one of a plurality of light sensing devices;

generating electrical signals, wherein each of the electrical signals isgenerated by a respective one of the light sensing devices, wherein theelectrical signal generated by each light sensing device representsintensity of the respective modulated light stream as a function oftime; and

acquiring from each of the light sensing devices a respective sequenceof samples of the respective electrical signal, wherein each samplesequence is usable to construct a respective sub-image representing arespective one of the regions of the incident light stream.

Computer System 3600

FIG. 36 illustrates one embodiment of a computer system 3600 that may beused to perform any of the method embodiments described herein, or, anycombination of the method embodiments described herein, or any subset ofany of the method embodiments described herein, or, any combination ofsuch subsets.

Computer system 3600 may include a processing unit 3610, a system memory3612, a set 3615 of one or more storage devices, a communication bus3620, a set 3625 of input devices, and a display system 3630.

System memory 3612 may include a set of semiconductor devices such asRAM devices (and perhaps also a set of ROM devices).

Storage devices 3615 may include any of various storage devices such asone or more memory media and/or memory access devices. For example,storage devices 3615 may include devices such as a CD/DVD-ROM drive, ahard disk, a magnetic disk drive, magnetic tape drives, etc.

Processing unit 3610 is configured to read and execute programinstructions, e.g., program instructions stored in system memory 3612and/or on one or more of the storage devices 3615. Processing unit 3610may couple to system memory 3612 through communication bus 3620 (orthrough a system of interconnected busses). The program instructionsconfigure the computer system 3600 to implement a method, e.g., any ofthe method embodiments described herein, or, any combination of themethod embodiments described herein, or, any subset of any of the methodembodiments described herein, or any combination of such subsets.

Processing unit 3610 may include one or more processors (e.g.,microprocessors).

One or more users may supply input to the computer system 3600 throughthe input devices 3625. Input devices 3625 may include devices such as akeyboard, a mouse, a touch-sensitive pad, a touch-sensitive screen, adrawing pad, a track ball, a light pen, a data glove, eye orientationand/or head orientation sensors, a microphone (or set of microphones),or any combination thereof

The display system 3630 may include any of a wide variety of displaydevices representing any of a wide variety of display technologies. Forexample, the display system may be a computer monitor, a head-mounteddisplay, a projector system,

a volumetric display, or a combination thereof. In some embodiments, thedisplay system may include a plurality of display devices. In oneembodiment, the display system may include a printer and/or a plotter.

In some embodiments, the computer system 3600 may include other devices,e.g., devices such as one or more graphics accelerators, one or morespeakers, a sound card, a video camera and a video card.

In some embodiments, computer system 3600 may include one or morecommunication devices 3635, e.g., a network interface card forinterfacing with a computer network.

The computer system may be configured with a software infrastructureincluding an operating system, one or more compilers for one or morecorresponding programming languages, and perhaps also one or moregraphics APIs (such as OpenGL®, Direct3D, Java 3D™). Any or all of thecompilers may be configured to perform expression rearrangementaccording to any or all of the method embodiments described herein.

In some embodiments, the computer system 3600 may be configured forcoupling to a compressive imaging system 3640, e.g., any of the varioussystem embodiments described herein. Computer system 3600 may beconfigured to receive samples from the compressive imaging system andoperate on the samples to produce images.

Any of the various embodiments described herein may be combined to formcomposite embodiments. Furthermore, any of the various embodimentsdescribed in U.S. Provisional Application No. 61/372,826, U.S.application Ser. No. 13/193,553 and U.S. application Ser. No. 13/193,556may be combined with any of the various embodiments described herein toform composite embodiments.

Although the embodiments above have been described in considerabledetail, numerous variations and modifications will become apparent tothose skilled in the art

once the above disclosure is fully appreciated. It is intended that thefollowing claims be interpreted to embrace all such variations andmodifications.

What is claimed is:
 1. A system comprising: a light modulation unitconfigured to modulate an incident stream of light with a sequence ofspatial patterns in order to produce a modulated light stream, whereinthe light modulation unit includes an array of light modulatingelements; a plurality of light sensing devices; an optical subsystemconfigured to deliver light from each of a plurality of spatial subsetsof the modulated light stream onto a respective one of the plurality oflight sensing devices, wherein the spatial subsets of the modulatedlight stream are produced by respective subsets of the array of lightmodulating elements, wherein each subset of the array of lightmodulating elements produces the respective spatial subset of themodulated light stream by modulating a respective spatial subset of theincident light stream, wherein the light sensing devices are configuredto generate respective electrical signals, wherein each of theelectrical signals represents intensity of the respective spatial subsetof the modulated light stream as a function of time; and a samplingsubsystem configured to acquire samples of the electrical signals,wherein the samples include sample subsets that correspond respectivelyto the electrical signals, wherein each sample subset includes aplurality of samples of the respective electrical signal; wherein thesampling subsystem is configured to acquire the samples of theelectrical signals in groups, wherein each of the groups includes atleast one sample of each of the electrical signals, wherein each of thegroups corresponds to a respective one of the spatial patterns; whereinsaid sampling subsystem is configured to acquire the groups of samplesof the electrical signals at a maximum group rate of R_(G) groups perunit time, wherein m_(SS) is the number of samples in each samplesubset, wherein N is the number of the light modulating elements,wherein n is less than or equal to N, wherein L is the number of thelight sensing devices, wherein a number m_(n/L) of the compressivesensing samples required to reconstruct an n/L-pixel image decreases asL increases, wherein L is selected so that m_(SS) is greater than orequal to m_(n/L), and, m_(SS)/R_(G) is less than or equal to a targetacquisition time.
 2. The system of claim 1, further comprising aprocessing unit configured to operate on each sample subset to constructthe respective sub-image.
 3. The system of claim 1, wherein the samplingsubsystem includes a plurality of analog-to-digital converters (ADCs),wherein each of the ADCs is configured to acquire the samples of theelectrical signal generated by a respective one of the light sensingdevices.
 4. The system of claim 3, wherein at least a first and a secondof the ADCs have different bit resolutions.
 5. The system of claim 1,wherein the optical subsystem includes a plurality of lenses, whereineach of the lenses is configured to direct the light from a respectiveone of the spatial subsets of the modulated light stream onto a lightsensing surface of a respective one of the light sensing devices.
 6. Thesystem of claim 1, wherein the optical subsystem includes a fibercoupling device and a plurality of fiber bundles, wherein the fibercoupling device is configured to couple the light from each of thespatial subsets of the modulated light stream onto a respective one ofthe fiber bundles.
 7. The system of claim 1, wherein the opticalsubsystem includes a plurality of light pipes whose internal surfacesare reflective.
 8. The system of claim 1, wherein the optical subsystemincludes a focusing lens, wherein the plurality of light sensing devicesare located at an image plane of the focusing lens.
 9. The system ofclaim 1, wherein the subsets of the array of the light modulatingelements are contiguous subsets.
 10. The system of claim 1, wherein atleast a first and a second of the light sensing devices have differentlight sensitivities.
 11. The system of claim 1, wherein the plurality oflight sensing devices are arranged in an array.
 12. The system of claim11, wherein the array is a two-dimensional rectangular array, whereinthe sampling subsystem includes one analog-to-digital converter percolumn of the array.
 13. The system of claim 1, wherein the plurality oflight sensing devices are incorporated into a first integrated circuit.14. The system of claim 13, wherein the sampling subsystem includes areadout integrated circuit, wherein the readout integrated circuit iscoupled to the first integrated circuit.
 15. The system of claim 14,wherein the readout integrated circuit is bump bonded to the firstintegrated circuit.
 16. A method comprising: modulating an incidentstream of light with a sequence of spatial patterns in order to producea modulated light stream, where said modulating is performed by an arrayof light modulating elements; delivering light from each of a pluralityof spatial subsets of the modulated light stream onto a respective oneof a plurality of light sensing devices, wherein the spatial subsets ofthe modulated light stream are produced by respective subsets of thearray of light modulating elements, wherein each subset of the array oflight modulating elements produces the respective spatial subset of themodulated light stream by modulating a respective spatial subset of theincident light stream; generating electrical signals, wherein each ofthe electrical signals is generated by a respective one of the lightsensing devices, wherein each of the electrical signals representsintensity of a respective one of the spatial subsets of the modulatedlight stream as a function of time; and acquiring samples of theelectrical signals, wherein the samples include sample subsets thatcorrespond respectively to the electrical signals, wherein each samplesubset includes a plurality of samples of the respective electricalsignal; wherein said acquiring the samples of the electrical signalsincludes acquiring the samples in groups, wherein each of the groupsincludes at least one sample of each of the electrical signals, whereineach of the groups corresponds to a respective one of the spatialpatterns; wherein the groups are acquired at a rate of R_(G) groups perunit time, wherein N is the number of the light modulating elements,wherein n is less than or equal to N, wherein L is the number of thelight sensing devices, wherein m_(SS) is the number of samples in eachsample subset, wherein a number m_(n/L) of the compressive sensingsamples required to reconstruct an n/L-pixel image decreases as Lincreases, wherein L is selected so that m_(SS) is greater than or equalto m_(n/L), and, m_(SS)/R_(G) is less than or equal to a targetacquisition time.
 17. The method of claim 16, further comprising:operating on the sample subsets to construct the respective sub-images.18. The method of claim 16, further comprising: transmitting the samplesubsets to a receiver through a communication channel.
 19. A systemcomprising: a light modulation unit configured to modulate an incidentstream of light with a sequence of spatial patterns in order to producea modulated light stream, wherein the light modulation unit includes anarray of light modulating elements; a plurality of light sensingdevices; an optical subsystem configured to direct spatial subsets ofthe modulated light stream to respective ones of the light sensingdevices, wherein the spatial subsets of the modulated light stream areproduced by respective non-overlapping regions of the array of lightmodulating elements, wherein each region of the array of lightmodulating elements produces the respective spatial subset of themodulated light stream by modulating a respective spatial subset of theincident light stream, wherein each of the light sensing devices isconfigured to generate a respective electrical signal that representsintensity of the respective spatial subset of the modulated light streamas a function of time; a plurality of analog-to-digital converters(ADCs), wherein each of the ADCs is configured to capture a respectivesequence of samples of the electrical signal generated by a respectiveone of the light sensing devices, wherein the sequence of samplescaptured by each of the ADCs is usable to construct a respectivesub-image that represents the corresponding spatial subset of theincident light stream; wherein the plurality of ADCs is configured toacquire the samples of the electrical signals in groups, wherein each ofthe groups includes at least one sample of each of the electricalsignals, wherein each of the groups corresponds to a respective one ofthe spatial patterns; wherein the plurality of ADCs is configured toacquire the groups of samples of the electrical signals at a maximumgroup rate of R_(G) groups per unit time, wherein m_(SS) is the numberof samples in each sequence of samples, wherein N is the number of thelight modulating elements, wherein n is less than or equal to N, whereinL is the number of the light sensing devices, wherein a number m_(n/L)of compressive sensing samples required to reconstruct an n/L-pixelimage decreases as L increases, wherein L is selected so that m_(SS) isgreater than or equal to m_(n/L), and, m_(SS)/R_(G) is less than orequal to a target acquisition time.
 20. The system of claim 19, whereinthe optical subsystem includes a plurality of lenses, wherein each ofthe lenses is configured to direct a respective one of the spatialsubsets of the modulated light stream onto a light sensing surface of arespective one of the light sensing devices.
 21. The system of claim 19,wherein the optical subsystem includes a fiber coupling device and aplurality of fiber bundles, wherein the fiber coupling device isconfigured to couple light from each of the spatial portions of themodulated light stream onto a respective one of the fiber bundles,wherein each of the fiber bundles is configured to deliver the lightfrom the respective spatial subset of the modulated light stream onto alight sensing surface of a respective one of the light sensing devices.22. The system of claim 19, wherein the plurality of light sensingdevices are incorporated into a single integrated circuit.
 23. Thesystem of claim 19, wherein the optical subsystem includes a focusinglens, wherein the plurality of light sensing devices are located at animage plane of the focusing lens.
 24. The system of claim 19, whereinthe plurality of ADCs are configured to capture their respective samplesequences synchronously.
 25. The system of claim 19, wherein theplurality of light sensing devices are arranged in an array.
 26. Thesystem of claim 19, wherein each of the sample sequences has a samplesize that is smaller than the number of pixels in the respectivesub-image.
 27. A system comprising: a light modulation unit configuredto modulate an incident stream of light with a sequence of spatialpatterns in order to produce a modulated light stream, wherein the lightmodulation unit includes an array of N light modulating elements; anarray of L light sensing elements, wherein L is greater than one butless than N; an optical subsystem configured to focus light from each ofa plurality of spatial portions of the modulated light stream onto arespective one of the array of light sensing elements, wherein thespatial portions of the modulated light stream are produced byrespective non-overlapping regions of the array of light modulatingelements, wherein each region of the array of light modulating elementsproduces the respective spatial portion of the modulated light stream bymodulating a respective spatial portion of the incident light stream,wherein the L light sensing elements are configured to generaterespective electrical signals, wherein each of the electrical signalsrepresents intensity of the respective spatial portion of the modulatedlight stream as a function of time; and circuitry configured to readgroups of samples from the array of L light sensing elements, whereineach group of samples includes L samples, wherein the L samples of eachgroup include one sample for each of the electrical signals; wherein thecircuitry is configured to read the groups at a maximum rate of R_(G)groups per unit time, wherein each of the groups corresponds to arespective one of the spatial patterns, wherein the sequence of spatialpatterns is configured so that subsets of the groups, each subset ofgroups being M groups in length, are usable to construct respectiveN-pixel images, wherein M is greater than one and less than or equal toN/L, wherein L is selected so that a rate of acquisition of the groupsubsets is greater than or equal to a target acquisition rate.
 28. Thesystem of claim 27, wherein the circuitry is included in a readoutintegrated circuit, wherein the array of L light sensing elements areincluded in a second integrated circuit.
 29. The system of claim 28,wherein the readout integrated circuit is bump bonded to the secondintegrated circuit.
 30. The system of claim 27, wherein the circuitryand the light sensing elements are included in a single integratedcircuit.
 31. A system comprising: a light modulation unit configured tomodulate an incident stream of light with a sequence of spatial patternsin order to produce a modulated light stream, wherein the lightmodulation unit includes an array of light modulating elements; aplurality of light sensing devices; an optical subsystem configured todeliver light from each of a plurality of spatial subsets of themodulated light stream onto a respective one of the plurality of lightsensing devices, wherein the spatial subsets of the modulated lightstream are produced by respective subsets of the array of lightmodulating elements, wherein each subset of the array of lightmodulating elements produces the respective spatial subset of themodulated light stream by modulating a respective spatial subset of theincident light stream, wherein the light sensing devices are configuredto generate respective electrical signals, wherein each of theelectrical signals represents intensity of the respective spatial subsetof the modulated light stream as a function of time; and a samplingsubsystem configured to acquire samples of the electrical signals,wherein the samples include sample subsets that correspond respectivelyto the electrical signals, wherein each sample subset includes aplurality of samples of the respective electrical signal; wherein eachof the sample subsets is usable to construct a respective sub-image ofan image, wherein each sub-image represents a respective one of thespatial subsets of the incident light stream; wherein N is the number ofthe light modulating elements, wherein L is the number of the lightsensing devices, wherein n is the number of pixels in said image,wherein n is less than or equal to N, wherein the light modulation unitis configured to modulate the incident light stream with the spatialpatterns at a maximum rate of R_(P) spatial patterns per unit time,wherein m_(SS) is the number of samples in each sample subset, wherein aminimum number m_(n/L) of compressive-sensing samples required toreconstruct an n/L-pixel image with given accuracy decreases as Lincreases, wherein m_(SS) and L are selected so that m_(SS) is greaterthan or equal to m_(n/L), and m_(SS)/R_(P) is less than or equal to atarget acquisition time.
 32. A system comprising: a light modulationunit configured to modulate an incident stream of light with a sequenceof spatial patterns in order to produce a modulated light stream,wherein the light modulation unit includes an array of light modulatingelements; a plurality of light sensing devices; an optical subsystemconfigured to deliver light from each of a plurality of spatial subsetsof the modulated light stream onto a respective one of the plurality oflight sensing devices, wherein the spatial subsets of the modulatedlight stream are produced by respective subsets of the array of lightmodulating elements, wherein each subset of the array of lightmodulating elements produces the respective spatial subset of themodulated light stream by modulating a respective spatial subset of theincident light stream, wherein the light sensing devices are configuredto generate respective electrical signals, wherein each of theelectrical signals represents intensity of the respective spatial subsetof the modulated light stream as a function of time; and a samplingsubsystem configured to acquire samples of the electrical signals,wherein the samples include sample subsets that correspond respectivelyto the electrical signals, wherein each sample subset includes aplurality of samples of the respective electrical signal; wherein eachof the sample subsets is usable to construct a respective sub-image ofan image, wherein each sub-image represents a respective one of thespatial subsets of the incident light stream; wherein N is the number ofthe light modulating elements, wherein L is the number of the lightsensing devices, wherein n is the number of pixels in said image,wherein n is less than or equal to N, wherein each sample subset isusable to construct the respective sub-image with n/L pixels, whereinthe sampling subsystem is configured so that the number of samples ineach sample subset is dynamically programmable to allow differenttradeoffs between image quality and acquisition time of the samples ofthe electrical signals.
 33. A method comprising: modulating an incidentstream of light with a sequence of spatial patterns in order to producea modulated light stream, where said modulating is performed by an arrayof light modulating elements; delivering light from each of a pluralityof spatial subsets of the modulated light stream onto a respective oneof a plurality of light sensing devices, wherein the spatial subsets ofthe modulated light stream are produced by respective subsets of thearray of light modulating elements, wherein each subset of the array oflight modulating elements produces the respective spatial subset of themodulated light stream by modulating a respective spatial subset of theincident light stream; generating electrical signals, wherein each ofthe electrical signals is generated by a respective one of the lightsensing devices, wherein each of the electrical signals representsintensity of a respective one of the spatial subsets of the modulatedlight stream as a function of time; and acquiring samples of theelectrical signals, wherein the samples include sample subsets thatcorrespond respectively to the electrical signals, wherein each samplesubset includes a plurality of samples of the respective electricalsignal; wherein each of the sample subsets is usable to construct arespective sub-image of an image, wherein each sub-image represents arespective one of the spatial subsets of the incident light stream;wherein N is the number of the light modulating elements, wherein L isthe number of the light sensing devices, wherein n is the number ofpixels in said image, wherein n is less than or equal to N, wherein theincident light stream is modulated with the spatial patterns at amaximum rate of R_(P) spatial patterns per unit time, wherein m_(SS) isthe number of samples in each sample subset, wherein a minimum numberm_(n/L) of compressive-sensing samples required to reconstruct ann/L-pixel image with given accuracy decreases as L increases, whereinm_(SS) and L are selected so that m_(SS) is greater than or equal tom_(n/L), and m_(SS)/R_(P) is less than or equal to a target acquisitiontime.
 34. A system comprising: a light modulation unit configured tomodulate an incident stream of light with a sequence of spatial patternsin order to produce a modulated light stream, wherein the lightmodulation unit includes an array of N light modulating elements; anarray of L light sensing elements, wherein L is greater than one butless than N; an optical subsystem configured to focus light from each ofa plurality of spatial portions of the modulated light stream onto arespective one of the array of light sensing elements, wherein thespatial portions of the modulated light stream are produced byrespective non-overlapping regions of the array of light modulatingelements, wherein each region of the array of light modulating elementsproduces the respective spatial portion of the modulated light stream bymodulating a respective spatial portion of the incident light stream,wherein the L light sensing elements are configured to generaterespective electrical signals, wherein each of the electrical signalsrepresents intensity of the respective spatial portion of the modulatedlight stream as a function of time; and circuitry configured to readgroups of samples from the array of L light sensing elements, whereineach group of samples includes L samples, wherein the L samples of eachgroup include one sample for each of the electrical signals; wherein thecircuitry is configured to read the groups at a maximum rate of R_(G)groups per unit time, wherein each of the groups corresponds to arespective one of the spatial patterns, wherein the sequence of spatialpatterns is configured so that M of said groups are usable to constructan N-pixel image representing the incident light stream, wherein M isgreater than one and less than or equal to N/L.
 35. The system of claim34, wherein L is selected to achieve an acquisition time for the Mgroups that is less than or equal to a target acquisition time.
 36. Thesystem of claim 34, wherein M is less than N/L.